May 27 21:03:30.462214 ip-10-0-128-93 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 27 21:03:30.462226 ip-10-0-128-93 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 27 21:03:30.462233 ip-10-0-128-93 systemd[1]: kubelet.service: Failed with result 'resources'.
May 27 21:03:30.462551 ip-10-0-128-93 systemd[1]: Failed to start Kubernetes Kubelet.
May 27 21:03:40.584962 ip-10-0-128-93 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 27 21:03:40.584976 ip-10-0-128-93 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d68dca59c2734ae6a55dd2edd4438b6f --
May 27 21:06:13.243899 ip-10-0-128-93 systemd[1]: Starting Kubernetes Kubelet...
May 27 21:06:13.658914 ip-10-0-128-93 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 27 21:06:13.658914 ip-10-0-128-93 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 27 21:06:13.658914 ip-10-0-128-93 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 27 21:06:13.658914 ip-10-0-128-93 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 27 21:06:13.658914 ip-10-0-128-93 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 27 21:06:13.660796 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.660684    2570 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 27 21:06:13.666764 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666743    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 27 21:06:13.666764 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666763    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666768    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666771    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666774    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666777    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666780    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666783    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666787    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666791    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666793    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666798    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666802    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666805    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666808    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666812    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666815    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666818    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666820    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666823    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666827    2570 feature_gate.go:328] unrecognized feature gate: Example
May 27 21:06:13.666823 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666830    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666833    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666836    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666839    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666842    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666845    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666847    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666850    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666852    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666855    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666857    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666860    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666862    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666865    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666867    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666870    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666873    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666876    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666879    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666881    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 27 21:06:13.667317 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666884    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666889    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666892    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666895    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666899    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666901    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666905    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666913    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666916    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666919    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666921    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666923    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666926    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666930    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666933    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666936    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666939    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666942    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666944    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 27 21:06:13.667905 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666947    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666950    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666952    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666955    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666957    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666960    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666963    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666965    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666968    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666970    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666973    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666976    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666979    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666982    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666985    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666987    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666990    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666993    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666995    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.666998    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 27 21:06:13.668369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667000    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667004    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667006    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667008    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667011    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667014    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667434    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667440    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667444    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667446    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667449    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667452    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667455    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667458    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667460    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667463    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667466    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667468    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667471    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667474    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 27 21:06:13.668908 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667476    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667480    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667482    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667485    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667488    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667491    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667494    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667496    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667499    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667502    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667504    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667507    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667509    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667512    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667515    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667518    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667521    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667523    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667526    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667528    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 27 21:06:13.669383 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667531    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667533    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667536    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667538    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667541    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667544    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667546    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667549    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667551    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667554    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667569    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667572    2570 feature_gate.go:328] unrecognized feature gate: Example
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667575    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667579    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667581    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667584    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667587    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667589    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667592    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667595    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 27 21:06:13.669902 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667597    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667601    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667604    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667607    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667610    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667612    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667616    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667620    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667624    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667627    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667630    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667633    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667636    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667639    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667641    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667644    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667650    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667653    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667656    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 27 21:06:13.670403 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667658    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667661    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667663    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667666    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667668    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667671    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667673    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667676    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667678    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667681    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667684    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667687    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.667690    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667769    2570 flags.go:64] FLAG: --address="0.0.0.0"
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667777    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667787    2570 flags.go:64] FLAG: --anonymous-auth="true"
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667792    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667797    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667800    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667805    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667811    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 27 21:06:13.670879 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667814    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667818    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667821    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667825    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667828    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667831    2570 flags.go:64] FLAG: --cgroup-root=""
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667834    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667838    2570 flags.go:64] FLAG: --client-ca-file=""
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667841    2570 flags.go:64] FLAG: --cloud-config=""
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667843    2570 flags.go:64] FLAG: --cloud-provider="external"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667846    2570 flags.go:64] FLAG: --cluster-dns="[]"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667851    2570 flags.go:64] FLAG: --cluster-domain=""
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667854    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667857    2570 flags.go:64] FLAG: --config-dir=""
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667860    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667863    2570 flags.go:64] FLAG: --container-log-max-files="5"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667867    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667870    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667874    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667877    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667881    2570 flags.go:64] FLAG: --contention-profiling="false"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667884    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667887    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667890    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667893    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 27 21:06:13.671401 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667898    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667901    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667904    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667907    2570 flags.go:64] FLAG: --enable-load-reader="false"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667910    2570 flags.go:64] FLAG: --enable-server="true"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667917    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667923    2570 flags.go:64] FLAG: --event-burst="100"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667926    2570 flags.go:64] FLAG: --event-qps="50"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667929    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667933    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667936    2570 flags.go:64] FLAG: --eviction-hard=""
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667939    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667942    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667945    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667948    2570 flags.go:64] FLAG: --eviction-soft=""
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667951    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667954    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667957    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667960    2570 flags.go:64] FLAG: --experimental-mounter-path=""
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667963    2570 flags.go:64] FLAG: --fail-cgroupv1="false"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667966    2570 flags.go:64] FLAG: --fail-swap-on="true"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667969    2570 flags.go:64] FLAG: --feature-gates=""
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667973    2570 flags.go:64] FLAG: --file-check-frequency="20s"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667976    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667980    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 27 21:06:13.672019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667983    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667986    2570 flags.go:64] FLAG: --healthz-port="10248"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667989    2570 flags.go:64] FLAG: --help="false"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667992    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-128-93.ec2.internal"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667995    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.667998    2570 flags.go:64] FLAG: --http-check-frequency="20s"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668002    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668006    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668009    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668012    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668015    2570 flags.go:64] FLAG: --image-service-endpoint=""
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668018    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668022    2570 flags.go:64] FLAG: --kube-api-burst="100"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668025    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668028    2570 flags.go:64] FLAG: --kube-api-qps="50"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668032    2570 flags.go:64] FLAG: --kube-reserved=""
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668035    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668038    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668041    2570 flags.go:64] FLAG: --kubelet-cgroups=""
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668044    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668047    2570 flags.go:64] FLAG: --lock-file=""
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668050    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668053    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668056    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 27 21:06:13.672635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668061    2570 flags.go:64] FLAG: --log-json-split-stream="false"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668064    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668067    2570 flags.go:64] FLAG: --log-text-split-stream="false"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668070    2570 flags.go:64] FLAG: --logging-format="text"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668073    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668076    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668079    2570 flags.go:64] FLAG: --manifest-url=""
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668082    2570 flags.go:64] FLAG: --manifest-url-header=""
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668087    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668092    2570 flags.go:64] FLAG: --max-open-files="1000000"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668096    2570 flags.go:64] FLAG: --max-pods="110"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668099    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668103    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668105    2570 flags.go:64] FLAG: --memory-manager-policy="None"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668109    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668112    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668114    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668117    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668125    2570 flags.go:64] FLAG: --node-status-max-images="50"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668129    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668133    2570 flags.go:64] FLAG: --oom-score-adj="-999"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668136    2570 flags.go:64] FLAG: --pod-cidr=""
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668139    2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 27 21:06:13.673215 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668145    2570 flags.go:64] FLAG: --pod-manifest-path=""
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668149    2570 flags.go:64] FLAG: --pod-max-pids="-1"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668152    2570 flags.go:64] FLAG: --pods-per-core="0"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668155    2570 flags.go:64] FLAG: --port="10250"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668158    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668161    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06fa8fa4a7c950a7d"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668164    2570 flags.go:64] FLAG: --qos-reserved=""
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668167    2570 flags.go:64] FLAG: --read-only-port="10255"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668170    2570 flags.go:64] FLAG: --register-node="true"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668173    2570 flags.go:64] FLAG: --register-schedulable="true"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668176    2570 flags.go:64] FLAG: --register-with-taints=""
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668180    2570 flags.go:64] FLAG: --registry-burst="10"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668183    2570 flags.go:64] FLAG: --registry-qps="5"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668186    2570 flags.go:64] FLAG: --reserved-cpus=""
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668189    2570 flags.go:64] FLAG: --reserved-memory=""
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668193    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668196    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668200    2570 flags.go:64] FLAG: --rotate-certificates="false"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668208    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668211    2570 flags.go:64] FLAG: --runonce="false"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668214    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668217    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668220    2570 flags.go:64] FLAG: --seccomp-default="false"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668223    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668226    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668229    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 27 21:06:13.673844 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668232    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668236    2570 flags.go:64] FLAG: --storage-driver-password="root"
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668239    2570 flags.go:64] FLAG: --storage-driver-secure="false"
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668242    2570 flags.go:64] FLAG: --storage-driver-table="stats"
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668246    2570 flags.go:64] FLAG: --storage-driver-user="root"
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668249    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668252    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668255    2570 flags.go:64] FLAG: --system-cgroups=""
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668259    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668264    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668267    2570 flags.go:64] FLAG: --tls-cert-file=""
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668270    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668275    2570 flags.go:64] FLAG: --tls-min-version=""
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668277    2570 flags.go:64] FLAG: --tls-private-key-file=""
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668280    2570 flags.go:64] FLAG: --topology-manager-policy="none"
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668283    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668286    2570 flags.go:64] FLAG: --topology-manager-scope="container"
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668289    2570 flags.go:64] FLAG: --v="2"
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668294    2570 flags.go:64] FLAG: --version="false"
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668297    2570 flags.go:64] FLAG: --vmodule=""
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668302    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668305    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668400    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668403    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 27 21:06:13.674517 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668409    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668412    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668415    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668417    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668420    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668423    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668425    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668428    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668431    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668433    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668436    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668438    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668442    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668445    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668448    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668450    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668453    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668456    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668458    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668461    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 27 21:06:13.675363 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668464    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668466    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668469    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668471    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668474    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668476    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668479    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668481    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668484    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668486    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668489    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668492    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668496    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668498    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668501    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668503    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668506    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668508    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668511    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668513    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 27 21:06:13.676023 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668516    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668519    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668521    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668524    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668528    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668531    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668533    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668536    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668538    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668541    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668544    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668546    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668549    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668551    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668554    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668569    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668572    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668575    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668577    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668580    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 27 21:06:13.676652 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668583    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668585    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668588    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668590    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668595    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668598    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668602    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668606    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668609    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668613    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668616    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668620    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668622    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668625    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668628    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668631    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668635    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668637    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668640    2570 feature_gate.go:328] unrecognized feature gate: Example
May 27 21:06:13.677156 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668643    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668646    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668648    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668651    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.668654    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.668663    2570 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.677286    2570 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.677305    2570 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677359    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677364    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677368    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677371    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677374    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677377    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677380    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677383    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 27 21:06:13.677645 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677386    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677389    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677392    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677395    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677397    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677400    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677403    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677405    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677408    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677411    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677415    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677417    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677420    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677423    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677425    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677429    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677431    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677434    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677438    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677441    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 27 21:06:13.678112 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677444    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677447    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677449    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677452    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677455    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677458    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677461    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677464    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677466    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677469    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677472    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677474    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677477    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677479    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677482    2570 feature_gate.go:328] unrecognized feature gate: Example
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677484    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677487    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677489    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677492    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677494    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 27 21:06:13.678618 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677497    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677500    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677502    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677505    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677508    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677510    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677513    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677515    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677517    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677520    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677523    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677526    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677529    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677531    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677534    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677537    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677539    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677542    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677544    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677547    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 27 21:06:13.679117 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677549    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677552    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677554    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677574    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677577    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677581    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677583    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677587    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677594    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677599    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677603    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677606    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677609    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677612    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677615    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677618    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677620    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 27 21:06:13.679635 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677623    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 27 21:06:13.680054 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.677628    2570 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 27 21:06:13.680054 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677752    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 27 21:06:13.680054 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677757    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 27 21:06:13.680054 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677761    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 27 21:06:13.680054 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677764    2570 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 27 21:06:13.680054 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677766    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 27 21:06:13.680054 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677769    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 27 21:06:13.680054 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677772    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 27 21:06:13.680054 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677775    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 27 21:06:13.680054 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677777    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
May 27 21:06:13.680054 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677780    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 27 21:06:13.680054 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677783    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 27 21:06:13.680054 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677786    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 27 21:06:13.680054 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677788    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 27 21:06:13.680054 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677791    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677793    2570 feature_gate.go:328] unrecognized feature gate: Example
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677796    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677799    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677801    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677804    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677807    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677810    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677812    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677816    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677819    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677821    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677824    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677827    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677830    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677833    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677835    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677837    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677840    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677843    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 27 21:06:13.680426 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677845    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677848    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677850    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677853    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677855    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677858    2570 feature_gate.go:328] unrecognized feature gate: Example2
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677860    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677863    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677865    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677868    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677871    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677873    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677875    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677878    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677880    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677883    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677885    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677888    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677890    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677894    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 27 21:06:13.680945 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677897    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677899    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677903    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677906    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677908    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677911    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677914    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677916    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677919    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677921    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677924    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677927    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677930    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677932    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677935    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677937    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677940    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677943    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677945    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 27 21:06:13.681470 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677948    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 27 21:06:13.682097 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677950    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 27 21:06:13.682097 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677953    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 27 21:06:13.682097 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677957    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 27 21:06:13.682097 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677960    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 27 21:06:13.682097 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677963    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 27 21:06:13.682097 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677966    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 27 21:06:13.682097 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677969    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 27 21:06:13.682097 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677971    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 27 21:06:13.682097 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677974    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 27 21:06:13.682097 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677977    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 27 21:06:13.682097 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677980    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 27 21:06:13.682097 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677984    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 27 21:06:13.682097 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:13.677987    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 27 21:06:13.682097 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.677992    2570 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 27 21:06:13.682097 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.678756    2570 server.go:962] "Client rotation is on, will bootstrap in background"
May 27 21:06:13.682526 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.682031    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 27 21:06:13.683103 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.683090    2570 server.go:1019] "Starting client certificate rotation"
May 27 21:06:13.683209 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.683189    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 27 21:06:13.683248 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.683232    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 27 21:06:13.709894 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.709869    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 27 21:06:13.711864 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.711842    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 27 21:06:13.730179 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.730150    2570 log.go:25] "Validated CRI v1 runtime API"
May 27 21:06:13.736661 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.736641    2570 log.go:25] "Validated CRI v1 image API"
May 27 21:06:13.738119 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.738101    2570 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 27 21:06:13.742850 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.742829    2570 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 27 21:06:13.743433 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.743409    2570 fs.go:135] Filesystem UUIDs: map[613055bb-cefc-49d9-b7e1-2d98241ac53d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f45de278-2eef-4d81-bd17-9ff1887608ba:/dev/nvme0n1p3]
May 27 21:06:13.743501 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.743437    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 27 21:06:13.748091 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.747964    2570 manager.go:217] Machine: {Timestamp:2026-05-27 21:06:13.747417423 +0000 UTC m=+0.386042579 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097412 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25d24a438117322eeed97cc144c199 SystemUUID:ec25d24a-4381-1732-2eee-d97cc144c199 BootID:d68dca59-c273-4ae6-a55d-d2edd4438b6f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:81:a4:62:ae:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:81:a4:62:ae:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:0a:ef:a9:55:4e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 27 21:06:13.749769 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.749754    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 27 21:06:13.749906 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.749887    2570 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 27 21:06:13.752145 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.752116    2570 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 27 21:06:13.752295 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.752147    2570 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-93.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 27 21:06:13.752344 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.752306    2570 topology_manager.go:138] "Creating topology manager with none policy"
May 27 21:06:13.752344 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.752314    2570 container_manager_linux.go:306] "Creating device plugin manager"
May 27 21:06:13.752344 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.752322    2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 27 21:06:13.752939 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.752929    2570 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 27 21:06:13.755034 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.755022    2570 state_mem.go:36] "Initialized new in-memory state store"
May 27 21:06:13.755155 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.755146    2570 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 27 21:06:13.757952 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.757941    2570 kubelet.go:491] "Attempting to sync node with API server"
May 27 21:06:13.757990 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.757957    2570 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 27 21:06:13.757990 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.757969    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 27 21:06:13.757990 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.757980    2570 kubelet.go:397] "Adding apiserver pod source"
May 27 21:06:13.757990 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.757989    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 27 21:06:13.759350 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.759333    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 27 21:06:13.759350 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.759354    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 27 21:06:13.763031 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.763005    2570 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 27 21:06:13.765378 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.765364    2570 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 27 21:06:13.767180 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.767167    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 27 21:06:13.767227 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.767185    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 27 21:06:13.767227 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.767192    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 27 21:06:13.767227 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.767198    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 27 21:06:13.767227 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.767226    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 27 21:06:13.767348 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.767233    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 27 21:06:13.767348 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.767246    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 27 21:06:13.767348 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.767252    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 27 21:06:13.767348 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.767259    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 27 21:06:13.767348 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.767265    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 27 21:06:13.767348 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.767278    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 27 21:06:13.767348 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.767284    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 27 21:06:13.768159 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.768149    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 27 21:06:13.768159 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.768160    2570 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 27 21:06:13.770868 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:13.770834    2570 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-93.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 27 21:06:13.770958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.770907    2570 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-93.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 27 21:06:13.771046 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:13.771011    2570 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 27 21:06:13.771886 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.771873    2570 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 27 21:06:13.771919 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.771913    2570 server.go:1295] "Started kubelet"
May 27 21:06:13.772033 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.771979    2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 27 21:06:13.772127 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.772024    2570 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 27 21:06:13.772127 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.772110    2570 server_v1.go:47] "podresources" method="list" useActivePods=true
May 27 21:06:13.772884 ip-10-0-128-93 systemd[1]: Started Kubernetes Kubelet.
May 27 21:06:13.773299 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.773239    2570 server.go:317] "Adding debug handlers to kubelet server"
May 27 21:06:13.773817 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.773733    2570 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 27 21:06:13.781592 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:13.781528    2570 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 27 21:06:13.782436 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.782417    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 27 21:06:13.783117 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.783105    2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 27 21:06:13.783200 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.783172    2570 volume_manager.go:295] "The desired_state_of_world populator starts"
May 27 21:06:13.783200 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.783180    2570 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 27 21:06:13.785505 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:13.785477    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-93.ec2.internal\" not found"
May 27 21:06:13.785606 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.785510    2570 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 27 21:06:13.785736 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.785720    2570 reconstruct.go:97] "Volume reconstruction finished"
May 27 21:06:13.785736 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.785735    2570 reconciler.go:26] "Reconciler: start to sync state"
May 27 21:06:13.786271 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.786252    2570 factory.go:55] Registering systemd factory
May 27 21:06:13.786342 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.786309    2570 factory.go:223] Registration of the systemd container factory successfully
May 27 21:06:13.787057 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.786662    2570 factory.go:153] Registering CRI-O factory
May 27 21:06:13.787057 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.786682    2570 factory.go:223] Registration of the crio container factory successfully
May 27 21:06:13.787057 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.786755    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 27 21:06:13.787057 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.786809    2570 factory.go:103] Registering Raw factory
May 27 21:06:13.787057 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.786834    2570 manager.go:1196] Started watching for new ooms in manager
May 27 21:06:13.787520 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.787449    2570 manager.go:319] Starting recovery of all containers
May 27 21:06:13.788770 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.788571    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cq6dj"
May 27 21:06:13.789213 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:13.789185    2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-93.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 27 21:06:13.792347 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:13.789050    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-93.ec2.internal.18b38825d2b78080  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-93.ec2.internal,UID:ip-10-0-128-93.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-93.ec2.internal,},FirstTimestamp:2026-05-27 21:06:13.77188672 +0000 UTC m=+0.410511876,LastTimestamp:2026-05-27 21:06:13.77188672 +0000 UTC m=+0.410511876,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-93.ec2.internal,}"
May 27 21:06:13.792460 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:13.792372    2570 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 27 21:06:13.794516 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.794492    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cq6dj"
May 27 21:06:13.800629 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.800606    2570 manager.go:324] Recovery completed
May 27 21:06:13.805198 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.805181    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 21:06:13.807850 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.807834    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasSufficientMemory"
May 27 21:06:13.807944 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.807870    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasNoDiskPressure"
May 27 21:06:13.807944 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.807885    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasSufficientPID"
May 27 21:06:13.808478 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.808466    2570 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 27 21:06:13.808478 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.808478    2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 27 21:06:13.808571 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.808494    2570 state_mem.go:36] "Initialized new in-memory state store"
May 27 21:06:13.809972 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:13.809903    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-93.ec2.internal.18b38825d4dc476f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-93.ec2.internal,UID:ip-10-0-128-93.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-93.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-93.ec2.internal,},FirstTimestamp:2026-05-27 21:06:13.807851375 +0000 UTC m=+0.446476535,LastTimestamp:2026-05-27 21:06:13.807851375 +0000 UTC m=+0.446476535,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-93.ec2.internal,}"
May 27 21:06:13.814659 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.814644    2570 policy_none.go:49] "None policy: Start"
May 27 21:06:13.814726 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.814664    2570 memory_manager.go:186] "Starting memorymanager" policy="None"
May 27 21:06:13.814726 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.814676    2570 state_mem.go:35] "Initializing new in-memory state store"
May 27 21:06:13.869860 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.863256    2570 manager.go:341] "Starting Device Plugin manager"
May 27 21:06:13.869860 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:13.863291    2570 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 27 21:06:13.869860 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.863301    2570 server.go:85] "Starting device plugin registration server"
May 27 21:06:13.869860 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.863608    2570 eviction_manager.go:189] "Eviction manager: starting control loop"
May 27 21:06:13.869860 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.863626    2570 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 27 21:06:13.869860 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.863705    2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 27 21:06:13.869860 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.863797    2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 27 21:06:13.869860 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.863808    2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 27 21:06:13.869860 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:13.864379    2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 27 21:06:13.869860 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:13.864412    2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-93.ec2.internal\" not found"
May 27 21:06:13.927885 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.927791    2570 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 27 21:06:13.929094 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.929076    2570 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 27 21:06:13.929204 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.929106    2570 status_manager.go:230] "Starting to sync pod status with apiserver"
May 27 21:06:13.929204 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.929132    2570 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 27 21:06:13.929204 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.929142    2570 kubelet.go:2451] "Starting kubelet main sync loop"
May 27 21:06:13.929204 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:13.929189    2570 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 27 21:06:13.938292 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.938271    2570 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 21:06:13.964506 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.964477    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 21:06:13.965949 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.965932    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasSufficientMemory"
May 27 21:06:13.966028 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.965961    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasNoDiskPressure"
May 27 21:06:13.966028 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.965972    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasSufficientPID"
May 27 21:06:13.966028 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.966010    2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-93.ec2.internal"
May 27 21:06:13.984907 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:13.984880    2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-93.ec2.internal"
May 27 21:06:13.984907 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:13.984909    2570 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-93.ec2.internal\": node \"ip-10-0-128-93.ec2.internal\" not found"
May 27 21:06:14.004899 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:14.004869    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-93.ec2.internal\" not found"
May 27 21:06:14.029981 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.029929    2570 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-93.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal"]
May 27 21:06:14.030144 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.030026    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 21:06:14.031904 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.031889    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasSufficientMemory"
May 27 21:06:14.031998 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.031916    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasNoDiskPressure"
May 27 21:06:14.031998 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.031925    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasSufficientPID"
May 27 21:06:14.033099 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.033084    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 21:06:14.033244 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.033220    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-93.ec2.internal"
May 27 21:06:14.033295 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.033261    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 21:06:14.034530 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.034514    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasSufficientMemory"
May 27 21:06:14.034624 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.034516    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasSufficientMemory"
May 27 21:06:14.034831 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.034806    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasNoDiskPressure"
May 27 21:06:14.034892 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.034833    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasSufficientPID"
May 27 21:06:14.034892 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.034539    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasNoDiskPressure"
May 27 21:06:14.034892 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.034870    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasSufficientPID"
May 27 21:06:14.036604 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.036194    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal"
May 27 21:06:14.036604 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.036242    2570 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 27 21:06:14.037538 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.037521    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasSufficientMemory"
May 27 21:06:14.037633 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.037553    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasNoDiskPressure"
May 27 21:06:14.037633 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.037588    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeHasSufficientPID"
May 27 21:06:14.066435 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:14.066406    2570 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-93.ec2.internal\" not found" node="ip-10-0-128-93.ec2.internal"
May 27 21:06:14.071124 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:14.071106    2570 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-93.ec2.internal\" not found" node="ip-10-0-128-93.ec2.internal"
May 27 21:06:14.086719 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.086693    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/96ba32301ea110b32229358267c0d072-config\") pod \"kube-apiserver-proxy-ip-10-0-128-93.ec2.internal\" (UID: \"96ba32301ea110b32229358267c0d072\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-93.ec2.internal"
May 27 21:06:14.086858 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.086727    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d3801e40acfa13e796e779dd0665dd4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal\" (UID: \"9d3801e40acfa13e796e779dd0665dd4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal"
May 27 21:06:14.086858 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.086755    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d3801e40acfa13e796e779dd0665dd4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal\" (UID: \"9d3801e40acfa13e796e779dd0665dd4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal"
May 27 21:06:14.105293 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:14.105263    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-93.ec2.internal\" not found"
May 27 21:06:14.187319 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.187256    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/96ba32301ea110b32229358267c0d072-config\") pod \"kube-apiserver-proxy-ip-10-0-128-93.ec2.internal\" (UID: \"96ba32301ea110b32229358267c0d072\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-93.ec2.internal"
May 27 21:06:14.187319 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.187290    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d3801e40acfa13e796e779dd0665dd4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal\" (UID: \"9d3801e40acfa13e796e779dd0665dd4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal"
May 27 21:06:14.187319 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.187317    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d3801e40acfa13e796e779dd0665dd4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal\" (UID: \"9d3801e40acfa13e796e779dd0665dd4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal"
May 27 21:06:14.187468 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.187357    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/96ba32301ea110b32229358267c0d072-config\") pod \"kube-apiserver-proxy-ip-10-0-128-93.ec2.internal\" (UID: \"96ba32301ea110b32229358267c0d072\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-93.ec2.internal"
May 27 21:06:14.187468 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.187373    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d3801e40acfa13e796e779dd0665dd4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal\" (UID: \"9d3801e40acfa13e796e779dd0665dd4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal"
May 27 21:06:14.187468 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.187415    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d3801e40acfa13e796e779dd0665dd4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal\" (UID: \"9d3801e40acfa13e796e779dd0665dd4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal"
May 27 21:06:14.206384 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:14.206352    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-93.ec2.internal\" not found"
May 27 21:06:14.307180 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:14.307144    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-93.ec2.internal\" not found"
May 27 21:06:14.370376 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.370348    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-93.ec2.internal"
May 27 21:06:14.375146 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.375124    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal"
May 27 21:06:14.408140 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:14.408104    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-93.ec2.internal\" not found"
May 27 21:06:14.508696 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:14.508603    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-93.ec2.internal\" not found"
May 27 21:06:14.609199 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:14.609158    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-93.ec2.internal\" not found"
May 27 21:06:14.682443 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.682407    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 27 21:06:14.683118 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.682622    2570 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 27 21:06:14.709827 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:14.709797    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-93.ec2.internal\" not found"
May 27 21:06:14.783356 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.783329    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 27 21:06:14.792878 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.792852    2570 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 27 21:06:14.797693 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.797658    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-26 21:01:13 +0000 UTC" deadline="2027-11-27 06:27:03.547646277 +0000 UTC"
May 27 21:06:14.797693 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.797688    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13161h20m48.749960954s"
May 27 21:06:14.810806 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:14.810776    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-93.ec2.internal\" not found"
May 27 21:06:14.818019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.817999    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-njwc5"
May 27 21:06:14.823292 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.823269    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-njwc5"
May 27 21:06:14.909055 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:14.909014    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3801e40acfa13e796e779dd0665dd4.slice/crio-e7276024cd720a3cb491aa425c7d7f82b65123edbc258d486f4f31ece941e3ae WatchSource:0}: Error finding container e7276024cd720a3cb491aa425c7d7f82b65123edbc258d486f4f31ece941e3ae: Status 404 returned error can't find the container with id e7276024cd720a3cb491aa425c7d7f82b65123edbc258d486f4f31ece941e3ae
May 27 21:06:14.910143 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:14.910110    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ba32301ea110b32229358267c0d072.slice/crio-c6d510ca8b51fab7ae7b8eeaeb64d8a6c00b971dd77c5c645ab27ec87dbfde0a WatchSource:0}: Error finding container c6d510ca8b51fab7ae7b8eeaeb64d8a6c00b971dd77c5c645ab27ec87dbfde0a: Status 404 returned error can't find the container with id c6d510ca8b51fab7ae7b8eeaeb64d8a6c00b971dd77c5c645ab27ec87dbfde0a
May 27 21:06:14.911191 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:14.911170    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-93.ec2.internal\" not found"
May 27 21:06:14.915963 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.915948    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 27 21:06:14.933231 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.933177    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal" event={"ID":"9d3801e40acfa13e796e779dd0665dd4","Type":"ContainerStarted","Data":"e7276024cd720a3cb491aa425c7d7f82b65123edbc258d486f4f31ece941e3ae"}
May 27 21:06:14.934267 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:14.934246    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-93.ec2.internal" event={"ID":"96ba32301ea110b32229358267c0d072","Type":"ContainerStarted","Data":"c6d510ca8b51fab7ae7b8eeaeb64d8a6c00b971dd77c5c645ab27ec87dbfde0a"}
May 27 21:06:15.011767 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:15.011730    2570 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-93.ec2.internal\" not found"
May 27 21:06:15.022311 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.022257    2570 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 21:06:15.037753 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.037721    2570 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 21:06:15.043600 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.043576    2570 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 21:06:15.086537 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.086508    2570 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-93.ec2.internal"
May 27 21:06:15.097521 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.097496    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 27 21:06:15.098630 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.098615    2570 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal"
May 27 21:06:15.110226 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.110206    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 27 21:06:15.760127 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.760070    2570 apiserver.go:52] "Watching apiserver"
May 27 21:06:15.772916 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.772889    2570 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 27 21:06:15.773415 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.773391    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-4g469","openshift-network-diagnostics/network-check-target-zvqrs","kube-system/konnectivity-agent-cwbhz","openshift-cluster-node-tuning-operator/tuned-vhwmz","openshift-dns/node-resolver-7cjkr","openshift-image-registry/node-ca-ktm2q","openshift-multus/multus-additional-cni-plugins-7lpb9","openshift-multus/network-metrics-daemon-nbt2c","openshift-network-operator/iptables-alerter-gfsbq","openshift-ovn-kubernetes/ovnkube-node-mwl2r","kube-system/kube-apiserver-proxy-ip-10-0-128-93.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal"]
May 27 21:06:15.777864 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.777834    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cwbhz"
May 27 21:06:15.778058 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.778029    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7cjkr"
May 27 21:06:15.780822 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.780793    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.781376 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.780953    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 27 21:06:15.781376 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.780996    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tnn7p\""
May 27 21:06:15.781376 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.781351    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wkn42\""
May 27 21:06:15.781376 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.781352    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 27 21:06:15.781648 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.781403    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 27 21:06:15.781757 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.781739    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 27 21:06:15.783024 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.783006    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.783720 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.783672    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 27 21:06:15.783720 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.783688    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 27 21:06:15.783881 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.783808    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 27 21:06:15.784719 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.784461    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 27 21:06:15.784719 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.784463    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-74lh2\""
May 27 21:06:15.784878 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.784849    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 27 21:06:15.786523 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.786502    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:15.786652 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:15.786627    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:15.786711 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.786656    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6l2vq\""
May 27 21:06:15.787287 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.787129    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 27 21:06:15.787287 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.787136    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 27 21:06:15.789379 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.789357    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ktm2q"
May 27 21:06:15.791613 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.791595    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4g469"
May 27 21:06:15.793073 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.793050    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 27 21:06:15.793169 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.793105    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 27 21:06:15.793169 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.793147    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 27 21:06:15.793279 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.793269    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f7wpc\""
May 27 21:06:15.793954 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.793913    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 27 21:06:15.793954 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.793929    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-p8q89\""
May 27 21:06:15.793954 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.793945    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:15.794135 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:15.794012    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:15.796058 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796039    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gfsbq"
May 27 21:06:15.796140 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796101    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztv9r\" (UniqueName: \"kubernetes.io/projected/a31a703f-7353-451c-9112-accce453fe8c-kube-api-access-ztv9r\") pod \"node-resolver-7cjkr\" (UID: \"a31a703f-7353-451c-9112-accce453fe8c\") " pod="openshift-dns/node-resolver-7cjkr"
May 27 21:06:15.796140 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796135    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-modprobe-d\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.796254 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796162    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgjgg\" (UniqueName: \"kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg\") pod \"network-check-target-zvqrs\" (UID: \"8d1f23cf-0c14-4a46-8314-85bc4b106ab4\") " pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:15.796254 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796193    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5e3d6151-d7e1-4794-b820-4cf1bd1c12d1-agent-certs\") pod \"konnectivity-agent-cwbhz\" (UID: \"5e3d6151-d7e1-4794-b820-4cf1bd1c12d1\") " pod="kube-system/konnectivity-agent-cwbhz"
May 27 21:06:15.796254 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796220    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5e3d6151-d7e1-4794-b820-4cf1bd1c12d1-konnectivity-ca\") pod \"konnectivity-agent-cwbhz\" (UID: \"5e3d6151-d7e1-4794-b820-4cf1bd1c12d1\") " pod="kube-system/konnectivity-agent-cwbhz"
May 27 21:06:15.796411 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796263    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/766a48c1-61da-40cc-be8b-1688536b1722-system-cni-dir\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.796411 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796321    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/766a48c1-61da-40cc-be8b-1688536b1722-os-release\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.796411 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796374    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-var-lib-kubelet\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.796411 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796406    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f550041-0005-439b-b856-db373e386c50-tmp\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.796661 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796440    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a31a703f-7353-451c-9112-accce453fe8c-hosts-file\") pod \"node-resolver-7cjkr\" (UID: \"a31a703f-7353-451c-9112-accce453fe8c\") " pod="openshift-dns/node-resolver-7cjkr"
May 27 21:06:15.796661 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796466    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87j5q\" (UniqueName: \"kubernetes.io/projected/7f550041-0005-439b-b856-db373e386c50-kube-api-access-87j5q\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.796661 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796510    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/766a48c1-61da-40cc-be8b-1688536b1722-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.796661 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796543    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/766a48c1-61da-40cc-be8b-1688536b1722-cnibin\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.796661 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796599    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/766a48c1-61da-40cc-be8b-1688536b1722-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.796661 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796627    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-kubernetes\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.796958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796666    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vd7w\" (UniqueName: \"kubernetes.io/projected/766a48c1-61da-40cc-be8b-1688536b1722-kube-api-access-5vd7w\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.796958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796697    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-sysconfig\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.796958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796737    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-systemd\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.796958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796755    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-sys\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.796958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796771    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31a703f-7353-451c-9112-accce453fe8c-tmp-dir\") pod \"node-resolver-7cjkr\" (UID: \"a31a703f-7353-451c-9112-accce453fe8c\") " pod="openshift-dns/node-resolver-7cjkr"
May 27 21:06:15.796958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796786    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/766a48c1-61da-40cc-be8b-1688536b1722-cni-binary-copy\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.796958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796811    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-sysctl-d\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.796958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796847    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-sysctl-conf\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.796958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796866    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-run\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.796958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796906    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-lib-modules\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.796958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796925    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-host\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.796958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796944    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f550041-0005-439b-b856-db373e386c50-etc-tuned\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.796958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.796963    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/766a48c1-61da-40cc-be8b-1688536b1722-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.798258 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.798231    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 27 21:06:15.798258 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.798252    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 27 21:06:15.798399 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.798310    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fwnbj\""
May 27 21:06:15.798458 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.798398    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 27 21:06:15.799181 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.799161    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.801480 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.801390    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 27 21:06:15.801593 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.801516    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:15.801865 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.801845    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 27 21:06:15.801949 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.801887    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6n97w\""
May 27 21:06:15.802034 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.802014    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 27 21:06:15.802111 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.802042    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 27 21:06:15.802158 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.802097    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 27 21:06:15.802198 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.802104    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 27 21:06:15.804634 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.804614    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 27 21:06:15.804746 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.804722    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rrrbw\""
May 27 21:06:15.805185 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.805168    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 27 21:06:15.805516 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.805498    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 27 21:06:15.824393 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.824309    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-26 21:01:14 +0000 UTC" deadline="2028-01-29 09:52:09.781808769 +0000 UTC"
May 27 21:06:15.824393 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.824356    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14676h45m53.957473213s"
May 27 21:06:15.825230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.825212    2570 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 27 21:06:15.886843 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.886806    2570 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 27 21:06:15.897369 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897335    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5e3d6151-d7e1-4794-b820-4cf1bd1c12d1-konnectivity-ca\") pod \"konnectivity-agent-cwbhz\" (UID: \"5e3d6151-d7e1-4794-b820-4cf1bd1c12d1\") " pod="kube-system/konnectivity-agent-cwbhz"
May 27 21:06:15.897369 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897371    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/766a48c1-61da-40cc-be8b-1688536b1722-system-cni-dir\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.897616 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897398    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/766a48c1-61da-40cc-be8b-1688536b1722-os-release\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.897616 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897425    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-var-lib-kubelet\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.897616 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897449    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f550041-0005-439b-b856-db373e386c50-tmp\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.897616 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897476    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/766a48c1-61da-40cc-be8b-1688536b1722-system-cni-dir\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.897616 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897517    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-var-lib-kubelet\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.897616 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897533    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/766a48c1-61da-40cc-be8b-1688536b1722-os-release\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.897616 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897578    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98aa4634-5b19-4d18-859e-457982a7dbb5-ovn-node-metrics-cert\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.897616 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897608    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a31a703f-7353-451c-9112-accce453fe8c-hosts-file\") pod \"node-resolver-7cjkr\" (UID: \"a31a703f-7353-451c-9112-accce453fe8c\") " pod="openshift-dns/node-resolver-7cjkr"
May 27 21:06:15.897967 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897632    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-87j5q\" (UniqueName: \"kubernetes.io/projected/7f550041-0005-439b-b856-db373e386c50-kube-api-access-87j5q\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.897967 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897687    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8qcc\" (UniqueName: \"kubernetes.io/projected/dddcd173-827c-4681-ab31-637bc3186afb-kube-api-access-d8qcc\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.897967 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897764    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a31a703f-7353-451c-9112-accce453fe8c-hosts-file\") pod \"node-resolver-7cjkr\" (UID: \"a31a703f-7353-451c-9112-accce453fe8c\") " pod="openshift-dns/node-resolver-7cjkr"
May 27 21:06:15.897967 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897837    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-run-systemd\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.897967 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897838    2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 27 21:06:15.897967 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897884    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-socket-dir\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:15.897967 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897910    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/766a48c1-61da-40cc-be8b-1688536b1722-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.897967 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897935    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/766a48c1-61da-40cc-be8b-1688536b1722-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.897967 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897963    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvd2r\" (UniqueName: \"kubernetes.io/projected/8627c382-6754-44fb-bb08-de5c1cd78407-kube-api-access-cvd2r\") pod \"node-ca-ktm2q\" (UID: \"8627c382-6754-44fb-bb08-de5c1cd78407\") " pod="openshift-image-registry/node-ca-ktm2q"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.897989    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-var-lib-cni-bin\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898012    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-etc-kubernetes\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898039    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b50ceea-19d6-4a3a-a409-46820b807d07-iptables-alerter-script\") pod \"iptables-alerter-gfsbq\" (UID: \"4b50ceea-19d6-4a3a-a409-46820b807d07\") " pod="openshift-network-operator/iptables-alerter-gfsbq"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898072    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-node-log\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898096    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898124    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-sysconfig\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898149    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-systemd\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898175    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-multus-cni-dir\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898200    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-os-release\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898226    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf6v5\" (UniqueName: \"kubernetes.io/projected/375b9e24-ceeb-431a-9102-ca5ac59437cb-kube-api-access-bf6v5\") pod \"network-metrics-daemon-nbt2c\" (UID: \"375b9e24-ceeb-431a-9102-ca5ac59437cb\") " pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898250    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b50ceea-19d6-4a3a-a409-46820b807d07-host-slash\") pod \"iptables-alerter-gfsbq\" (UID: \"4b50ceea-19d6-4a3a-a409-46820b807d07\") " pod="openshift-network-operator/iptables-alerter-gfsbq"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898277    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-cni-bin\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898275    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5e3d6151-d7e1-4794-b820-4cf1bd1c12d1-konnectivity-ca\") pod \"konnectivity-agent-cwbhz\" (UID: \"5e3d6151-d7e1-4794-b820-4cf1bd1c12d1\") " pod="kube-system/konnectivity-agent-cwbhz"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898299    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-systemd-units\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898325    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/766a48c1-61da-40cc-be8b-1688536b1722-cni-binary-copy\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.898364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898352    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-lib-modules\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898369    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-sysconfig\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898376    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-host\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898415    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-cnibin\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898422    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-host\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898443    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-multus-socket-dir-parent\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898490    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-systemd\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898498    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/766a48c1-61da-40cc-be8b-1688536b1722-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898491    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/766a48c1-61da-40cc-be8b-1688536b1722-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898552    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-var-lib-kubelet\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898613    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-etc-openvswitch\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898637    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-run-openvswitch\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898647    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-lib-modules\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898668    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/766a48c1-61da-40cc-be8b-1688536b1722-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898713    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ztv9r\" (UniqueName: \"kubernetes.io/projected/a31a703f-7353-451c-9112-accce453fe8c-kube-api-access-ztv9r\") pod \"node-resolver-7cjkr\" (UID: \"a31a703f-7353-451c-9112-accce453fe8c\") " pod="openshift-dns/node-resolver-7cjkr"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898735    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs\") pod \"network-metrics-daemon-nbt2c\" (UID: \"375b9e24-ceeb-431a-9102-ca5ac59437cb\") " pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898751    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-sys-fs\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:15.899165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898769    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-modprobe-d\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898786    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-log-socket\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898796    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/766a48c1-61da-40cc-be8b-1688536b1722-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898802    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898850    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5e3d6151-d7e1-4794-b820-4cf1bd1c12d1-agent-certs\") pod \"konnectivity-agent-cwbhz\" (UID: \"5e3d6151-d7e1-4794-b820-4cf1bd1c12d1\") " pod="kube-system/konnectivity-agent-cwbhz"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898866    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/766a48c1-61da-40cc-be8b-1688536b1722-cni-binary-copy\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898878    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-hostroot\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898902    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-modprobe-d\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898913    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-kubelet\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898939    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98aa4634-5b19-4d18-859e-457982a7dbb5-ovnkube-config\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.898965    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-registration-dir\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899008    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899047    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-multus-conf-dir\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899084    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-run-multus-certs\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899108    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9czm\" (UniqueName: \"kubernetes.io/projected/4b50ceea-19d6-4a3a-a409-46820b807d07-kube-api-access-n9czm\") pod \"iptables-alerter-gfsbq\" (UID: \"4b50ceea-19d6-4a3a-a409-46820b807d07\") " pod="openshift-network-operator/iptables-alerter-gfsbq"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899132    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-run-netns\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.899988 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899159    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/766a48c1-61da-40cc-be8b-1688536b1722-cnibin\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899184    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-kubernetes\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899207    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-system-cni-dir\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dddcd173-827c-4681-ab31-637bc3186afb-cni-binary-copy\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899254    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-slash\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899270    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-kubernetes\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899277    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-run-ovn\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899305    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5vd7w\" (UniqueName: \"kubernetes.io/projected/766a48c1-61da-40cc-be8b-1688536b1722-kube-api-access-5vd7w\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899332    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-sys\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899336    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/766a48c1-61da-40cc-be8b-1688536b1722-cnibin\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899356    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8627c382-6754-44fb-bb08-de5c1cd78407-host\") pod \"node-ca-ktm2q\" (UID: \"8627c382-6754-44fb-bb08-de5c1cd78407\") " pod="openshift-image-registry/node-ca-ktm2q"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899381    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dddcd173-827c-4681-ab31-637bc3186afb-multus-daemon-config\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899402    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-sys\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899407    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98aa4634-5b19-4d18-859e-457982a7dbb5-env-overrides\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899432    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-run-netns\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899461    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31a703f-7353-451c-9112-accce453fe8c-tmp-dir\") pod \"node-resolver-7cjkr\" (UID: \"a31a703f-7353-451c-9112-accce453fe8c\") " pod="openshift-dns/node-resolver-7cjkr"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899582    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-sysctl-d\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.900748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899608    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-sysctl-conf\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899632    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-run\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899656    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f550041-0005-439b-b856-db373e386c50-etc-tuned\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899680    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98aa4634-5b19-4d18-859e-457982a7dbb5-ovnkube-script-lib\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899702    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8627c382-6754-44fb-bb08-de5c1cd78407-serviceca\") pod \"node-ca-ktm2q\" (UID: \"8627c382-6754-44fb-bb08-de5c1cd78407\") " pod="openshift-image-registry/node-ca-ktm2q"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899724    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-run-k8s-cni-cncf-io\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899752    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-var-lib-cni-multus\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899777    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-var-lib-openvswitch\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899781    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-run\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899800    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899812    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31a703f-7353-451c-9112-accce453fe8c-tmp-dir\") pod \"node-resolver-7cjkr\" (UID: \"a31a703f-7353-451c-9112-accce453fe8c\") " pod="openshift-dns/node-resolver-7cjkr"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899824    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-device-dir\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899846    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrzm9\" (UniqueName: \"kubernetes.io/projected/d2381928-8d26-438e-96e5-bac8f26e07d0-kube-api-access-jrzm9\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899874    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgjgg\" (UniqueName: \"kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg\") pod \"network-check-target-zvqrs\" (UID: \"8d1f23cf-0c14-4a46-8314-85bc4b106ab4\") " pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899877    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-sysctl-d\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899902    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-cni-netd\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899926    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zff\" (UniqueName: \"kubernetes.io/projected/98aa4634-5b19-4d18-859e-457982a7dbb5-kube-api-access-r5zff\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:15.901450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.899957    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7f550041-0005-439b-b856-db373e386c50-etc-sysctl-conf\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.902163 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.901489    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f550041-0005-439b-b856-db373e386c50-tmp\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.902163 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.901980    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5e3d6151-d7e1-4794-b820-4cf1bd1c12d1-agent-certs\") pod \"konnectivity-agent-cwbhz\" (UID: \"5e3d6151-d7e1-4794-b820-4cf1bd1c12d1\") " pod="kube-system/konnectivity-agent-cwbhz"
May 27 21:06:15.902163 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.902029    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7f550041-0005-439b-b856-db373e386c50-etc-tuned\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.914922 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:15.914880    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 21:06:15.914922 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:15.914923    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 21:06:15.915117 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:15.914937    2570 projected.go:194] Error preparing data for projected volume kube-api-access-jgjgg for pod openshift-network-diagnostics/network-check-target-zvqrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 21:06:15.915117 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:15.915041    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg podName:8d1f23cf-0c14-4a46-8314-85bc4b106ab4 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:16.415005847 +0000 UTC m=+3.053631004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jgjgg" (UniqueName: "kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg") pod "network-check-target-zvqrs" (UID: "8d1f23cf-0c14-4a46-8314-85bc4b106ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 21:06:15.917255 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.917227    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztv9r\" (UniqueName: \"kubernetes.io/projected/a31a703f-7353-451c-9112-accce453fe8c-kube-api-access-ztv9r\") pod \"node-resolver-7cjkr\" (UID: \"a31a703f-7353-451c-9112-accce453fe8c\") " pod="openshift-dns/node-resolver-7cjkr"
May 27 21:06:15.917375 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.917230    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vd7w\" (UniqueName: \"kubernetes.io/projected/766a48c1-61da-40cc-be8b-1688536b1722-kube-api-access-5vd7w\") pod \"multus-additional-cni-plugins-7lpb9\" (UID: \"766a48c1-61da-40cc-be8b-1688536b1722\") " pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:15.917375 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.917352    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87j5q\" (UniqueName: \"kubernetes.io/projected/7f550041-0005-439b-b856-db373e386c50-kube-api-access-87j5q\") pod \"tuned-vhwmz\" (UID: \"7f550041-0005-439b-b856-db373e386c50\") " pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:15.940046 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.940014    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-5ws56"]
May 27 21:06:15.944349 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:15.944328    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:15.944475 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:15.944417    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:16.000934 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.000891    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98aa4634-5b19-4d18-859e-457982a7dbb5-ovn-node-metrics-cert\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.001127 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.000941    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d8qcc\" (UniqueName: \"kubernetes.io/projected/dddcd173-827c-4681-ab31-637bc3186afb-kube-api-access-d8qcc\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.001127 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.000965    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-run-systemd\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.001127 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.000990    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-socket-dir\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:16.001127 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001057    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-run-systemd\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.001127 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001112    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cvd2r\" (UniqueName: \"kubernetes.io/projected/8627c382-6754-44fb-bb08-de5c1cd78407-kube-api-access-cvd2r\") pod \"node-ca-ktm2q\" (UID: \"8627c382-6754-44fb-bb08-de5c1cd78407\") " pod="openshift-image-registry/node-ca-ktm2q"
May 27 21:06:16.001375 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001141    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-var-lib-cni-bin\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.001375 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001141    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-socket-dir\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:16.001375 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001167    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-etc-kubernetes\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.001375 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001216    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-etc-kubernetes\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.001375 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001246    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-var-lib-cni-bin\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.001375 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001324    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b50ceea-19d6-4a3a-a409-46820b807d07-iptables-alerter-script\") pod \"iptables-alerter-gfsbq\" (UID: \"4b50ceea-19d6-4a3a-a409-46820b807d07\") " pod="openshift-network-operator/iptables-alerter-gfsbq"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001379    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-node-log\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001407    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001435    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-multus-cni-dir\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001445    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-node-log\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001459    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-os-release\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001485    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001499    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bf6v5\" (UniqueName: \"kubernetes.io/projected/375b9e24-ceeb-431a-9102-ca5ac59437cb-kube-api-access-bf6v5\") pod \"network-metrics-daemon-nbt2c\" (UID: \"375b9e24-ceeb-431a-9102-ca5ac59437cb\") " pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001526    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-os-release\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001528    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b50ceea-19d6-4a3a-a409-46820b807d07-host-slash\") pod \"iptables-alerter-gfsbq\" (UID: \"4b50ceea-19d6-4a3a-a409-46820b807d07\") " pod="openshift-network-operator/iptables-alerter-gfsbq"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001579    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-multus-cni-dir\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001579    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b50ceea-19d6-4a3a-a409-46820b807d07-host-slash\") pod \"iptables-alerter-gfsbq\" (UID: \"4b50ceea-19d6-4a3a-a409-46820b807d07\") " pod="openshift-network-operator/iptables-alerter-gfsbq"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001580    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-cni-bin\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001629    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0216b723-92f1-4f86-a216-384ab8ea7891-dbus\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001638    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-cni-bin\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001656    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-systemd-units\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.001688 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001687    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-systemd-units\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001731    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-cnibin\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001761    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-multus-socket-dir-parent\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001790    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-var-lib-kubelet\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001818    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-etc-openvswitch\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001828    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-cnibin\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001858    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b50ceea-19d6-4a3a-a409-46820b807d07-iptables-alerter-script\") pod \"iptables-alerter-gfsbq\" (UID: \"4b50ceea-19d6-4a3a-a409-46820b807d07\") " pod="openshift-network-operator/iptables-alerter-gfsbq"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001873    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-run-openvswitch\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001872    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-var-lib-kubelet\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001889    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-etc-openvswitch\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001913    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-multus-socket-dir-parent\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001930    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs\") pod \"network-metrics-daemon-nbt2c\" (UID: \"375b9e24-ceeb-431a-9102-ca5ac59437cb\") " pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001960    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-sys-fs\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001933    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-run-openvswitch\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.001996    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-log-socket\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:16.002022    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002055    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-sys-fs\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002072    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.002408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002097    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-log-socket\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:16.002106    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs podName:375b9e24-ceeb-431a-9102-ca5ac59437cb nodeName:}" failed. No retries permitted until 2026-05-27 21:06:16.502084597 +0000 UTC m=+3.140709755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs") pod "network-metrics-daemon-nbt2c" (UID: "375b9e24-ceeb-431a-9102-ca5ac59437cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002024    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002146    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-hostroot\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002164    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-kubelet\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002188    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98aa4634-5b19-4d18-859e-457982a7dbb5-ovnkube-config\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002211    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-registration-dir\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002235    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002242    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-kubelet\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002260    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-multus-conf-dir\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002306    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002310    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-multus-conf-dir\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002306    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-hostroot\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002349    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-registration-dir\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002381    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-run-multus-certs\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002411    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n9czm\" (UniqueName: \"kubernetes.io/projected/4b50ceea-19d6-4a3a-a409-46820b807d07-kube-api-access-n9czm\") pod \"iptables-alerter-gfsbq\" (UID: \"4b50ceea-19d6-4a3a-a409-46820b807d07\") " pod="openshift-network-operator/iptables-alerter-gfsbq"
May 27 21:06:16.003230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002436    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-run-netns\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002457    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-run-multus-certs\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002463    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-system-cni-dir\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002487    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dddcd173-827c-4681-ab31-637bc3186afb-cni-binary-copy\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002511    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-slash\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002506    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-run-netns\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002537    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-run-ovn\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002585    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0216b723-92f1-4f86-a216-384ab8ea7891-kubelet-config\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002609    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002621    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-system-cni-dir\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002636    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8627c382-6754-44fb-bb08-de5c1cd78407-host\") pod \"node-ca-ktm2q\" (UID: \"8627c382-6754-44fb-bb08-de5c1cd78407\") " pod="openshift-image-registry/node-ca-ktm2q"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002659    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dddcd173-827c-4681-ab31-637bc3186afb-multus-daemon-config\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002684    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98aa4634-5b19-4d18-859e-457982a7dbb5-env-overrides\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002706    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-run-netns\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002719    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-slash\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002736    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98aa4634-5b19-4d18-859e-457982a7dbb5-ovnkube-script-lib\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002763    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8627c382-6754-44fb-bb08-de5c1cd78407-serviceca\") pod \"node-ca-ktm2q\" (UID: \"8627c382-6754-44fb-bb08-de5c1cd78407\") " pod="openshift-image-registry/node-ca-ktm2q"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002784    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-run-netns\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.003893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002788    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-run-k8s-cni-cncf-io\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002805    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8627c382-6754-44fb-bb08-de5c1cd78407-host\") pod \"node-ca-ktm2q\" (UID: \"8627c382-6754-44fb-bb08-de5c1cd78407\") " pod="openshift-image-registry/node-ca-ktm2q"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002726    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-run-ovn\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002828    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-run-k8s-cni-cncf-io\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002829    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-var-lib-cni-multus\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002869    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-var-lib-openvswitch\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002896    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002924    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-device-dir\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002954    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jrzm9\" (UniqueName: \"kubernetes.io/projected/d2381928-8d26-438e-96e5-bac8f26e07d0-kube-api-access-jrzm9\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.003032    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-cni-netd\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.003062    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zff\" (UniqueName: \"kubernetes.io/projected/98aa4634-5b19-4d18-859e-457982a7dbb5-kube-api-access-r5zff\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.003954    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98aa4634-5b19-4d18-859e-457982a7dbb5-ovnkube-script-lib\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.004026    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dddcd173-827c-4681-ab31-637bc3186afb-multus-daemon-config\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.004064    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-device-dir\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.004132    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dddcd173-827c-4681-ab31-637bc3186afb-cni-binary-copy\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.004158    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8627c382-6754-44fb-bb08-de5c1cd78407-serviceca\") pod \"node-ca-ktm2q\" (UID: \"8627c382-6754-44fb-bb08-de5c1cd78407\") " pod="openshift-image-registry/node-ca-ktm2q"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.004219    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2381928-8d26-438e-96e5-bac8f26e07d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:16.004680 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.004228    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-host-cni-netd\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.005408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002871    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dddcd173-827c-4681-ab31-637bc3186afb-host-var-lib-cni-multus\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.005408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.002799    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98aa4634-5b19-4d18-859e-457982a7dbb5-ovnkube-config\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.005408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.004280    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98aa4634-5b19-4d18-859e-457982a7dbb5-var-lib-openvswitch\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.005408 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.004653    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98aa4634-5b19-4d18-859e-457982a7dbb5-ovn-node-metrics-cert\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.007603 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.007580    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98aa4634-5b19-4d18-859e-457982a7dbb5-env-overrides\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.010822 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.010762    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9czm\" (UniqueName: \"kubernetes.io/projected/4b50ceea-19d6-4a3a-a409-46820b807d07-kube-api-access-n9czm\") pod \"iptables-alerter-gfsbq\" (UID: \"4b50ceea-19d6-4a3a-a409-46820b807d07\") " pod="openshift-network-operator/iptables-alerter-gfsbq"
May 27 21:06:16.010968 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.010950    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf6v5\" (UniqueName: \"kubernetes.io/projected/375b9e24-ceeb-431a-9102-ca5ac59437cb-kube-api-access-bf6v5\") pod \"network-metrics-daemon-nbt2c\" (UID: \"375b9e24-ceeb-431a-9102-ca5ac59437cb\") " pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:16.011240 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.011217    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8qcc\" (UniqueName: \"kubernetes.io/projected/dddcd173-827c-4681-ab31-637bc3186afb-kube-api-access-d8qcc\") pod \"multus-4g469\" (UID: \"dddcd173-827c-4681-ab31-637bc3186afb\") " pod="openshift-multus/multus-4g469"
May 27 21:06:16.011382 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.011365    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvd2r\" (UniqueName: \"kubernetes.io/projected/8627c382-6754-44fb-bb08-de5c1cd78407-kube-api-access-cvd2r\") pod \"node-ca-ktm2q\" (UID: \"8627c382-6754-44fb-bb08-de5c1cd78407\") " pod="openshift-image-registry/node-ca-ktm2q"
May 27 21:06:16.014666 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.014642    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrzm9\" (UniqueName: \"kubernetes.io/projected/d2381928-8d26-438e-96e5-bac8f26e07d0-kube-api-access-jrzm9\") pod \"aws-ebs-csi-driver-node-pprds\" (UID: \"d2381928-8d26-438e-96e5-bac8f26e07d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:16.014919 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.014899    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zff\" (UniqueName: \"kubernetes.io/projected/98aa4634-5b19-4d18-859e-457982a7dbb5-kube-api-access-r5zff\") pod \"ovnkube-node-mwl2r\" (UID: \"98aa4634-5b19-4d18-859e-457982a7dbb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.090457 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.090422    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cwbhz"
May 27 21:06:16.097191 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.097162    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7cjkr"
May 27 21:06:16.104989 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.104963    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0216b723-92f1-4f86-a216-384ab8ea7891-kubelet-config\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:16.105134 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.104994    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:16.105134 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.105043    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0216b723-92f1-4f86-a216-384ab8ea7891-dbus\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:16.105134 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.105094    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0216b723-92f1-4f86-a216-384ab8ea7891-kubelet-config\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:16.105243 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.105201    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0216b723-92f1-4f86-a216-384ab8ea7891-dbus\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:16.105243 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:16.105203    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 27 21:06:16.105310 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:16.105282    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret podName:0216b723-92f1-4f86-a216-384ab8ea7891 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:16.60526061 +0000 UTC m=+3.243885759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret") pod "global-pull-secret-syncer-5ws56" (UID: "0216b723-92f1-4f86-a216-384ab8ea7891") : object "kube-system"/"original-pull-secret" not registered
May 27 21:06:16.106943 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.106925    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7lpb9"
May 27 21:06:16.112008 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.111983    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vhwmz"
May 27 21:06:16.118644 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.118618    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ktm2q"
May 27 21:06:16.125361 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.125338    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4g469"
May 27 21:06:16.132957 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.132933    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gfsbq"
May 27 21:06:16.139627 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.139604    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:16.145335 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.145311    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds"
May 27 21:06:16.508538 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.508496    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs\") pod \"network-metrics-daemon-nbt2c\" (UID: \"375b9e24-ceeb-431a-9102-ca5ac59437cb\") " pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:16.508742 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.508577    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgjgg\" (UniqueName: \"kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg\") pod \"network-check-target-zvqrs\" (UID: \"8d1f23cf-0c14-4a46-8314-85bc4b106ab4\") " pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:16.508742 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:16.508669    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 21:06:16.508742 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:16.508694    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 21:06:16.508742 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:16.508713    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 21:06:16.508742 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:16.508729    2570 projected.go:194] Error preparing data for projected volume kube-api-access-jgjgg for pod openshift-network-diagnostics/network-check-target-zvqrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 21:06:16.508742 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:16.508746    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs podName:375b9e24-ceeb-431a-9102-ca5ac59437cb nodeName:}" failed. No retries permitted until 2026-05-27 21:06:17.508729845 +0000 UTC m=+4.147355009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs") pod "network-metrics-daemon-nbt2c" (UID: "375b9e24-ceeb-431a-9102-ca5ac59437cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 21:06:16.508998 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:16.508777    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg podName:8d1f23cf-0c14-4a46-8314-85bc4b106ab4 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:17.508762322 +0000 UTC m=+4.147387471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgjgg" (UniqueName: "kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg") pod "network-check-target-zvqrs" (UID: "8d1f23cf-0c14-4a46-8314-85bc4b106ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 21:06:16.561968 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:16.561936    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8627c382_6754_44fb_bb08_de5c1cd78407.slice/crio-b9653a1a64c811602d6da60e776d56a80126609036f72b8784764d73076d118c WatchSource:0}: Error finding container b9653a1a64c811602d6da60e776d56a80126609036f72b8784764d73076d118c: Status 404 returned error can't find the container with id b9653a1a64c811602d6da60e776d56a80126609036f72b8784764d73076d118c
May 27 21:06:16.562981 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:16.562950    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f550041_0005_439b_b856_db373e386c50.slice/crio-b1b4d39f92cb98d14f10e971bee4296ec7a95912198bf55412873d78cbf50b3a WatchSource:0}: Error finding container b1b4d39f92cb98d14f10e971bee4296ec7a95912198bf55412873d78cbf50b3a: Status 404 returned error can't find the container with id b1b4d39f92cb98d14f10e971bee4296ec7a95912198bf55412873d78cbf50b3a
May 27 21:06:16.563641 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:16.563614    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766a48c1_61da_40cc_be8b_1688536b1722.slice/crio-a06bdf8c85302dff1e0ae1990f8bcc112725ed020f6b010d0198e8d2ac6fadc2 WatchSource:0}: Error finding container a06bdf8c85302dff1e0ae1990f8bcc112725ed020f6b010d0198e8d2ac6fadc2: Status 404 returned error can't find the container with id a06bdf8c85302dff1e0ae1990f8bcc112725ed020f6b010d0198e8d2ac6fadc2
May 27 21:06:16.565361 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:16.564932    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3d6151_d7e1_4794_b820_4cf1bd1c12d1.slice/crio-80e1b72e5b58bbd2b33b7dd10b4ac18a096b243a40387e5a46efa0b5649266ae WatchSource:0}: Error finding container 80e1b72e5b58bbd2b33b7dd10b4ac18a096b243a40387e5a46efa0b5649266ae: Status 404 returned error can't find the container with id 80e1b72e5b58bbd2b33b7dd10b4ac18a096b243a40387e5a46efa0b5649266ae
May 27 21:06:16.567387 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:16.567364    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98aa4634_5b19_4d18_859e_457982a7dbb5.slice/crio-d5ff06cde308618f644186b2a83b58a408ec4af0e4c1929ec6a81acc136dd607 WatchSource:0}: Error finding container d5ff06cde308618f644186b2a83b58a408ec4af0e4c1929ec6a81acc136dd607: Status 404 returned error can't find the container with id d5ff06cde308618f644186b2a83b58a408ec4af0e4c1929ec6a81acc136dd607
May 27 21:06:16.568311 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:16.568290    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddddcd173_827c_4681_ab31_637bc3186afb.slice/crio-811b0353dcf11926c944406020b0bf90395f1bf07e3205cef58af8fb7bcc9fff WatchSource:0}: Error finding container 811b0353dcf11926c944406020b0bf90395f1bf07e3205cef58af8fb7bcc9fff: Status 404 returned error can't find the container with id 811b0353dcf11926c944406020b0bf90395f1bf07e3205cef58af8fb7bcc9fff
May 27 21:06:16.568846 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:16.568823    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b50ceea_19d6_4a3a_a409_46820b807d07.slice/crio-0a4bcca4b96ecf64a3f75c23c9115dece44b5b056fcbd753eec676c96f53dfb9 WatchSource:0}: Error finding container 0a4bcca4b96ecf64a3f75c23c9115dece44b5b056fcbd753eec676c96f53dfb9: Status 404 returned error can't find the container with id 0a4bcca4b96ecf64a3f75c23c9115dece44b5b056fcbd753eec676c96f53dfb9
May 27 21:06:16.589266 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:16.589235    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2381928_8d26_438e_96e5_bac8f26e07d0.slice/crio-2260fe2377073842cf8531a5682850e044bdba80a1c79abebb0383cffb8fb765 WatchSource:0}: Error finding container 2260fe2377073842cf8531a5682850e044bdba80a1c79abebb0383cffb8fb765: Status 404 returned error can't find the container with id 2260fe2377073842cf8531a5682850e044bdba80a1c79abebb0383cffb8fb765
May 27 21:06:16.590264 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:16.590233    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31a703f_7353_451c_9112_accce453fe8c.slice/crio-056dfc24ec61ccb07aac4e013080600b6a8fa9c9ac64a7eeab132968ea598318 WatchSource:0}: Error finding container 056dfc24ec61ccb07aac4e013080600b6a8fa9c9ac64a7eeab132968ea598318: Status 404 returned error can't find the container with id 056dfc24ec61ccb07aac4e013080600b6a8fa9c9ac64a7eeab132968ea598318
May 27 21:06:16.609870 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.609697    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:16.609949 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:16.609826    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 27 21:06:16.609985 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:16.609966    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret podName:0216b723-92f1-4f86-a216-384ab8ea7891 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:17.609946368 +0000 UTC m=+4.248571516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret") pod "global-pull-secret-syncer-5ws56" (UID: "0216b723-92f1-4f86-a216-384ab8ea7891") : object "kube-system"/"original-pull-secret" not registered
May 27 21:06:16.824910 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.824803    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-26 21:01:14 +0000 UTC" deadline="2028-01-26 07:57:06.420739889 +0000 UTC"
May 27 21:06:16.824910 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.824848    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14602h50m49.595905709s"
May 27 21:06:16.929810 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.929788    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:16.929914 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:16.929895    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:16.939018 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.938984    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-93.ec2.internal" event={"ID":"96ba32301ea110b32229358267c0d072","Type":"ContainerStarted","Data":"ac4d6628dc9973e2567765ea547f1fe2368992f1b705cfb61f408ee3ed92b241"}
May 27 21:06:16.939848 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.939824    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds" event={"ID":"d2381928-8d26-438e-96e5-bac8f26e07d0","Type":"ContainerStarted","Data":"2260fe2377073842cf8531a5682850e044bdba80a1c79abebb0383cffb8fb765"}
May 27 21:06:16.940920 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.940893    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r" event={"ID":"98aa4634-5b19-4d18-859e-457982a7dbb5","Type":"ContainerStarted","Data":"d5ff06cde308618f644186b2a83b58a408ec4af0e4c1929ec6a81acc136dd607"}
May 27 21:06:16.944265 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.944241    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cwbhz" event={"ID":"5e3d6151-d7e1-4794-b820-4cf1bd1c12d1","Type":"ContainerStarted","Data":"80e1b72e5b58bbd2b33b7dd10b4ac18a096b243a40387e5a46efa0b5649266ae"}
May 27 21:06:16.945570 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.945535    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7lpb9" event={"ID":"766a48c1-61da-40cc-be8b-1688536b1722","Type":"ContainerStarted","Data":"a06bdf8c85302dff1e0ae1990f8bcc112725ed020f6b010d0198e8d2ac6fadc2"}
May 27 21:06:16.946483 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.946456    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ktm2q" event={"ID":"8627c382-6754-44fb-bb08-de5c1cd78407","Type":"ContainerStarted","Data":"b9653a1a64c811602d6da60e776d56a80126609036f72b8784764d73076d118c"}
May 27 21:06:16.947861 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.947837    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal" event={"ID":"9d3801e40acfa13e796e779dd0665dd4","Type":"ContainerStarted","Data":"daeab7fedb5fe1ed469cea6a131de56301e21360b0ca09975c3021934ccbe395"}
May 27 21:06:16.948838 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.948812    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7cjkr" event={"ID":"a31a703f-7353-451c-9112-accce453fe8c","Type":"ContainerStarted","Data":"056dfc24ec61ccb07aac4e013080600b6a8fa9c9ac64a7eeab132968ea598318"}
May 27 21:06:16.949754 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.949733    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4g469" event={"ID":"dddcd173-827c-4681-ab31-637bc3186afb","Type":"ContainerStarted","Data":"811b0353dcf11926c944406020b0bf90395f1bf07e3205cef58af8fb7bcc9fff"}
May 27 21:06:16.950602 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.950581    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gfsbq" event={"ID":"4b50ceea-19d6-4a3a-a409-46820b807d07","Type":"ContainerStarted","Data":"0a4bcca4b96ecf64a3f75c23c9115dece44b5b056fcbd753eec676c96f53dfb9"}
May 27 21:06:16.951405 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.951385    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vhwmz" event={"ID":"7f550041-0005-439b-b856-db373e386c50","Type":"ContainerStarted","Data":"b1b4d39f92cb98d14f10e971bee4296ec7a95912198bf55412873d78cbf50b3a"}
May 27 21:06:16.964364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:16.964309    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-93.ec2.internal" podStartSLOduration=1.964293097 podStartE2EDuration="1.964293097s" podCreationTimestamp="2026-05-27 21:06:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 21:06:16.962645952 +0000 UTC m=+3.601271139" watchObservedRunningTime="2026-05-27 21:06:16.964293097 +0000 UTC m=+3.602918262"
May 27 21:06:17.515577 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:17.515497    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs\") pod \"network-metrics-daemon-nbt2c\" (UID: \"375b9e24-ceeb-431a-9102-ca5ac59437cb\") " pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:17.515782 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:17.515631    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgjgg\" (UniqueName: \"kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg\") pod \"network-check-target-zvqrs\" (UID: \"8d1f23cf-0c14-4a46-8314-85bc4b106ab4\") " pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:17.515864 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:17.515801    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 21:06:17.515864 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:17.515829    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 21:06:17.515864 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:17.515843    2570 projected.go:194] Error preparing data for projected volume kube-api-access-jgjgg for pod openshift-network-diagnostics/network-check-target-zvqrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 21:06:17.515993 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:17.515906    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg podName:8d1f23cf-0c14-4a46-8314-85bc4b106ab4 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:19.515885692 +0000 UTC m=+6.154510840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgjgg" (UniqueName: "kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg") pod "network-check-target-zvqrs" (UID: "8d1f23cf-0c14-4a46-8314-85bc4b106ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 21:06:17.516363 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:17.516341    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 21:06:17.516466 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:17.516394    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs podName:375b9e24-ceeb-431a-9102-ca5ac59437cb nodeName:}" failed. No retries permitted until 2026-05-27 21:06:19.516379859 +0000 UTC m=+6.155005016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs") pod "network-metrics-daemon-nbt2c" (UID: "375b9e24-ceeb-431a-9102-ca5ac59437cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 21:06:17.616425 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:17.616383    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:17.616661 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:17.616571    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 27 21:06:17.616661 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:17.616636    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret podName:0216b723-92f1-4f86-a216-384ab8ea7891 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:19.616617197 +0000 UTC m=+6.255242353 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret") pod "global-pull-secret-syncer-5ws56" (UID: "0216b723-92f1-4f86-a216-384ab8ea7891") : object "kube-system"/"original-pull-secret" not registered
May 27 21:06:17.930747 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:17.930716    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:17.931197 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:17.930841    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:17.931197 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:17.931188    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:17.931315 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:17.931280    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:17.958667 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:17.958626    2570 generic.go:358] "Generic (PLEG): container finished" podID="9d3801e40acfa13e796e779dd0665dd4" containerID="daeab7fedb5fe1ed469cea6a131de56301e21360b0ca09975c3021934ccbe395" exitCode=0
May 27 21:06:17.959753 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:17.959726    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal" event={"ID":"9d3801e40acfa13e796e779dd0665dd4","Type":"ContainerDied","Data":"daeab7fedb5fe1ed469cea6a131de56301e21360b0ca09975c3021934ccbe395"}
May 27 21:06:17.959871 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:17.959765    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal" event={"ID":"9d3801e40acfa13e796e779dd0665dd4","Type":"ContainerStarted","Data":"bcee5bfd7cb229b2f0bdefc9b5feef249ab8739d573f8ace2da29f6e29366be3"}
May 27 21:06:18.930271 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:18.930234    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:18.930450 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:18.930387    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:19.536994 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:19.536157    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs\") pod \"network-metrics-daemon-nbt2c\" (UID: \"375b9e24-ceeb-431a-9102-ca5ac59437cb\") " pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:19.536994 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:19.536245    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgjgg\" (UniqueName: \"kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg\") pod \"network-check-target-zvqrs\" (UID: \"8d1f23cf-0c14-4a46-8314-85bc4b106ab4\") " pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:19.536994 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:19.536388    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 21:06:19.536994 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:19.536407    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 21:06:19.536994 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:19.536420    2570 projected.go:194] Error preparing data for projected volume kube-api-access-jgjgg for pod openshift-network-diagnostics/network-check-target-zvqrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 21:06:19.536994 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:19.536481    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg podName:8d1f23cf-0c14-4a46-8314-85bc4b106ab4 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:23.53646012 +0000 UTC m=+10.175085277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgjgg" (UniqueName: "kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg") pod "network-check-target-zvqrs" (UID: "8d1f23cf-0c14-4a46-8314-85bc4b106ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 21:06:19.536994 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:19.536901    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 21:06:19.536994 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:19.536951    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs podName:375b9e24-ceeb-431a-9102-ca5ac59437cb nodeName:}" failed. No retries permitted until 2026-05-27 21:06:23.536936229 +0000 UTC m=+10.175561388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs") pod "network-metrics-daemon-nbt2c" (UID: "375b9e24-ceeb-431a-9102-ca5ac59437cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 21:06:19.637172 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:19.637027    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:19.637347 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:19.637204    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 27 21:06:19.637347 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:19.637276    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret podName:0216b723-92f1-4f86-a216-384ab8ea7891 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:23.637257914 +0000 UTC m=+10.275883070 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret") pod "global-pull-secret-syncer-5ws56" (UID: "0216b723-92f1-4f86-a216-384ab8ea7891") : object "kube-system"/"original-pull-secret" not registered
May 27 21:06:19.929716 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:19.929683    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:19.929888 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:19.929736    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:19.929888 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:19.929814    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:19.930270 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:19.930242    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:20.930545 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:20.929768    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:20.930545 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:20.930066    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:21.929416 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:21.929379    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:21.929589 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:21.929379    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:21.929589 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:21.929518    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:21.929589 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:21.929548    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:22.929519 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:22.929482    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:22.929975 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:22.929652    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:23.572412 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:23.571759    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs\") pod \"network-metrics-daemon-nbt2c\" (UID: \"375b9e24-ceeb-431a-9102-ca5ac59437cb\") " pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:23.572412 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:23.571845    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgjgg\" (UniqueName: \"kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg\") pod \"network-check-target-zvqrs\" (UID: \"8d1f23cf-0c14-4a46-8314-85bc4b106ab4\") " pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:23.572412 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:23.572002    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 21:06:23.572412 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:23.572018    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 21:06:23.572412 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:23.572032    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 21:06:23.572412 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:23.572045    2570 projected.go:194] Error preparing data for projected volume kube-api-access-jgjgg for pod openshift-network-diagnostics/network-check-target-zvqrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 21:06:23.572412 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:23.572067    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs podName:375b9e24-ceeb-431a-9102-ca5ac59437cb nodeName:}" failed. No retries permitted until 2026-05-27 21:06:31.572047957 +0000 UTC m=+18.210673119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs") pod "network-metrics-daemon-nbt2c" (UID: "375b9e24-ceeb-431a-9102-ca5ac59437cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 21:06:23.572412 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:23.572084    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg podName:8d1f23cf-0c14-4a46-8314-85bc4b106ab4 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:31.572074897 +0000 UTC m=+18.210700047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgjgg" (UniqueName: "kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg") pod "network-check-target-zvqrs" (UID: "8d1f23cf-0c14-4a46-8314-85bc4b106ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 21:06:23.673132 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:23.673089    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:23.673334 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:23.673203    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 27 21:06:23.673334 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:23.673276    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret podName:0216b723-92f1-4f86-a216-384ab8ea7891 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:31.673255576 +0000 UTC m=+18.311880723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret") pod "global-pull-secret-syncer-5ws56" (UID: "0216b723-92f1-4f86-a216-384ab8ea7891") : object "kube-system"/"original-pull-secret" not registered
May 27 21:06:23.930851 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:23.930817    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:23.931320 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:23.930934    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:23.931320 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:23.931052    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:23.931320 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:23.931133    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:24.929922 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:24.929797    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:24.930115 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:24.929947    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:25.930296 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:25.929798    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:25.930296 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:25.929935    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:25.930296 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:25.929998    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:25.930296 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:25.930124    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:26.929577 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:26.929534    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:26.929753 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:26.929695    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:27.929569 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:27.929525    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:27.930065 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:27.929536    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:27.930065 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:27.929717    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:27.930065 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:27.929869    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:28.929348 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:28.929317    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:28.929495 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:28.929453    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:29.929430 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:29.929391    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:29.929885 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:29.929447    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:29.929885 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:29.929530    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:29.929885 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:29.929678    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:30.929690 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:30.929652    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:30.930147 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:30.929806    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:31.636172 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:31.636120    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgjgg\" (UniqueName: \"kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg\") pod \"network-check-target-zvqrs\" (UID: \"8d1f23cf-0c14-4a46-8314-85bc4b106ab4\") " pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:31.636372 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:31.636209    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs\") pod \"network-metrics-daemon-nbt2c\" (UID: \"375b9e24-ceeb-431a-9102-ca5ac59437cb\") " pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:31.636372 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:31.636323    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 21:06:31.636372 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:31.636327    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 21:06:31.636372 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:31.636356    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 21:06:31.636372 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:31.636368    2570 projected.go:194] Error preparing data for projected volume kube-api-access-jgjgg for pod openshift-network-diagnostics/network-check-target-zvqrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 21:06:31.636635 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:31.636394    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs podName:375b9e24-ceeb-431a-9102-ca5ac59437cb nodeName:}" failed. No retries permitted until 2026-05-27 21:06:47.636372902 +0000 UTC m=+34.274998050 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs") pod "network-metrics-daemon-nbt2c" (UID: "375b9e24-ceeb-431a-9102-ca5ac59437cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 21:06:31.636635 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:31.636419    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg podName:8d1f23cf-0c14-4a46-8314-85bc4b106ab4 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:47.636402747 +0000 UTC m=+34.275027903 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgjgg" (UniqueName: "kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg") pod "network-check-target-zvqrs" (UID: "8d1f23cf-0c14-4a46-8314-85bc4b106ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 21:06:31.737019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:31.736976    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:31.737207 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:31.737144    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 27 21:06:31.737261 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:31.737224    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret podName:0216b723-92f1-4f86-a216-384ab8ea7891 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:47.737207539 +0000 UTC m=+34.375832683 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret") pod "global-pull-secret-syncer-5ws56" (UID: "0216b723-92f1-4f86-a216-384ab8ea7891") : object "kube-system"/"original-pull-secret" not registered
May 27 21:06:31.929740 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:31.929646    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:31.930189 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:31.929646    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:31.930189 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:31.930145    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:31.930308 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:31.930265    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:32.929776 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:32.929738    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:32.930257 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:32.929881    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:33.930465 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:33.930431    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:33.930956 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:33.930543    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:33.930956 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:33.930583    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:33.930956 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:33.930688    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:34.929768 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:34.929433    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:34.929935 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:34.929852    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:34.992823 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:34.992791    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds" event={"ID":"d2381928-8d26-438e-96e5-bac8f26e07d0","Type":"ContainerStarted","Data":"e03194a6d2d84aa511ea4e97f46d994b41867a531d4a2cfe1092a7490b5d97ea"}
May 27 21:06:34.994622 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:34.994593    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r" event={"ID":"98aa4634-5b19-4d18-859e-457982a7dbb5","Type":"ContainerStarted","Data":"cda38f8a6b67b77ea7ec3e610558c7d510fd5736a2cf7bb2a5b90e284fcef272"}
May 27 21:06:34.994736 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:34.994634    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r" event={"ID":"98aa4634-5b19-4d18-859e-457982a7dbb5","Type":"ContainerStarted","Data":"4557595b46168c01a9c52e9190bfcc667fcd1c5e73825b2e1bb8f61d144b0afa"}
May 27 21:06:34.996015 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:34.995962    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cwbhz" event={"ID":"5e3d6151-d7e1-4794-b820-4cf1bd1c12d1","Type":"ContainerStarted","Data":"5f32d088994de44533764cd74caaffea105e6e7fdb8479ce6f3092ec5cc7465f"}
May 27 21:06:34.997489 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:34.997468    2570 generic.go:358] "Generic (PLEG): container finished" podID="766a48c1-61da-40cc-be8b-1688536b1722" containerID="97e3f1c4efaf2e1dd132844993d8bc54e898c7803d0e81440eb3463f9ad353db" exitCode=0
May 27 21:06:34.997587 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:34.997525    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7lpb9" event={"ID":"766a48c1-61da-40cc-be8b-1688536b1722","Type":"ContainerDied","Data":"97e3f1c4efaf2e1dd132844993d8bc54e898c7803d0e81440eb3463f9ad353db"}
May 27 21:06:34.999106 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:34.998901    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ktm2q" event={"ID":"8627c382-6754-44fb-bb08-de5c1cd78407","Type":"ContainerStarted","Data":"778b0a840ded4497a8a59719a0d809fc12a330eb053ec5b66f8f128c146dbc80"}
May 27 21:06:35.000364 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:35.000329    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7cjkr" event={"ID":"a31a703f-7353-451c-9112-accce453fe8c","Type":"ContainerStarted","Data":"a5c0bce63d34675d246abf76ed0e2c5cc45c3fdc515cde225f96500240af1545"}
May 27 21:06:35.002011 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:35.001842    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4g469" event={"ID":"dddcd173-827c-4681-ab31-637bc3186afb","Type":"ContainerStarted","Data":"1a5bd777d4bd6264cfbd833a6a114f821c3b4f6ff91ef7ab3f3d8c61698b0382"}
May 27 21:06:35.003315 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:35.003289    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vhwmz" event={"ID":"7f550041-0005-439b-b856-db373e386c50","Type":"ContainerStarted","Data":"8efd9095e1167a89ea0e76488d54d47b4ac1867c4e0d4c0e63e47d85fba7f00d"}
May 27 21:06:35.014602 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:35.014501    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-93.ec2.internal" podStartSLOduration=20.014484664 podStartE2EDuration="20.014484664s" podCreationTimestamp="2026-05-27 21:06:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 21:06:17.974895854 +0000 UTC m=+4.613521047" watchObservedRunningTime="2026-05-27 21:06:35.014484664 +0000 UTC m=+21.653109828"
May 27 21:06:35.030275 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:35.030234    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ktm2q" podStartSLOduration=7.827156794 podStartE2EDuration="21.030218593s" podCreationTimestamp="2026-05-27 21:06:14 +0000 UTC" firstStartedPulling="2026-05-27 21:06:16.564768116 +0000 UTC m=+3.203393263" lastFinishedPulling="2026-05-27 21:06:29.767829905 +0000 UTC m=+16.406455062" observedRunningTime="2026-05-27 21:06:35.030216585 +0000 UTC m=+21.668841761" watchObservedRunningTime="2026-05-27 21:06:35.030218593 +0000 UTC m=+21.668843758"
May 27 21:06:35.030834 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:35.030804    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cwbhz" podStartSLOduration=4.088341324 podStartE2EDuration="22.030795977s" podCreationTimestamp="2026-05-27 21:06:13 +0000 UTC" firstStartedPulling="2026-05-27 21:06:16.566509687 +0000 UTC m=+3.205134844" lastFinishedPulling="2026-05-27 21:06:34.508964338 +0000 UTC m=+21.147589497" observedRunningTime="2026-05-27 21:06:35.014081212 +0000 UTC m=+21.652706377" watchObservedRunningTime="2026-05-27 21:06:35.030795977 +0000 UTC m=+21.669421141"
May 27 21:06:35.044296 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:35.044249    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-4g469" podStartSLOduration=3.082619215 podStartE2EDuration="21.044229747s" podCreationTimestamp="2026-05-27 21:06:14 +0000 UTC" firstStartedPulling="2026-05-27 21:06:16.588066636 +0000 UTC m=+3.226691779" lastFinishedPulling="2026-05-27 21:06:34.549677156 +0000 UTC m=+21.188302311" observedRunningTime="2026-05-27 21:06:35.044007875 +0000 UTC m=+21.682633039" watchObservedRunningTime="2026-05-27 21:06:35.044229747 +0000 UTC m=+21.682854914"
May 27 21:06:35.058402 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:35.058352    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-7cjkr" podStartSLOduration=4.141138312 podStartE2EDuration="22.058333294s" podCreationTimestamp="2026-05-27 21:06:13 +0000 UTC" firstStartedPulling="2026-05-27 21:06:16.591779336 +0000 UTC m=+3.230404481" lastFinishedPulling="2026-05-27 21:06:34.508974314 +0000 UTC m=+21.147599463" observedRunningTime="2026-05-27 21:06:35.057814335 +0000 UTC m=+21.696439502" watchObservedRunningTime="2026-05-27 21:06:35.058333294 +0000 UTC m=+21.696958459"
May 27 21:06:35.089627 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:35.089586    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vhwmz" podStartSLOduration=3.145481314 podStartE2EDuration="21.089570091s" podCreationTimestamp="2026-05-27 21:06:14 +0000 UTC" firstStartedPulling="2026-05-27 21:06:16.564874102 +0000 UTC m=+3.203499256" lastFinishedPulling="2026-05-27 21:06:34.508962873 +0000 UTC m=+21.147588033" observedRunningTime="2026-05-27 21:06:35.08896039 +0000 UTC m=+21.727585555" watchObservedRunningTime="2026-05-27 21:06:35.089570091 +0000 UTC m=+21.728195247"
May 27 21:06:35.804894 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:35.804851    2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 27 21:06:35.875609 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:35.875500    2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-27T21:06:35.804875506Z","UUID":"ce50988e-0c08-4dc6-aba2-6f444a9b7ddd","Handler":null,"Name":"","Endpoint":""}
May 27 21:06:35.878397 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:35.878372    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 27 21:06:35.878397 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:35.878401    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 27 21:06:35.932302 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:35.932273    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:35.932461 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:35.932273    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:35.932461 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:35.932372    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:35.932461 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:35.932443    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:36.007446 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:36.007404    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds" event={"ID":"d2381928-8d26-438e-96e5-bac8f26e07d0","Type":"ContainerStarted","Data":"fb64e5f8846ad0eac16ddce8d6b5e75dcbb3acec9074e9451a63e91911380bae"}
May 27 21:06:36.010293 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:36.010262    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r" event={"ID":"98aa4634-5b19-4d18-859e-457982a7dbb5","Type":"ContainerStarted","Data":"d5db2f4725ad67b66da2c8542de952fab60e33d92f170b5d4eb857023124c251"}
May 27 21:06:36.010438 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:36.010300    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r" event={"ID":"98aa4634-5b19-4d18-859e-457982a7dbb5","Type":"ContainerStarted","Data":"2beadbb65db1ecffa02dea57033794cddd21cf36e5d18347a12a8ec4242f84d7"}
May 27 21:06:36.010438 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:36.010316    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r" event={"ID":"98aa4634-5b19-4d18-859e-457982a7dbb5","Type":"ContainerStarted","Data":"6acba1656b96727bcc18b046789d2add56beeb4d39f8cdeb8861746542ce625f"}
May 27 21:06:36.010438 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:36.010328    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r" event={"ID":"98aa4634-5b19-4d18-859e-457982a7dbb5","Type":"ContainerStarted","Data":"c468456034fb98f3687c6e8347e0002f1a658f9cccc5179ea4e805c571033258"}
May 27 21:06:36.328696 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:36.328615    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cwbhz"
May 27 21:06:36.329412 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:36.329308    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cwbhz"
May 27 21:06:36.929394 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:36.929360    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:36.929590 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:36.929492    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:37.014870 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:37.014816    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds" event={"ID":"d2381928-8d26-438e-96e5-bac8f26e07d0","Type":"ContainerStarted","Data":"c3f87f92187c57314d455eb2e04f13a2f1a33f58edf5d622215cb987778386b4"}
May 27 21:06:37.016314 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:37.016285    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gfsbq" event={"ID":"4b50ceea-19d6-4a3a-a409-46820b807d07","Type":"ContainerStarted","Data":"f59f284049abe5239c3374230a7c5656fb6dc096259470b296a691405ab5071f"}
May 27 21:06:37.016728 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:37.016710    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cwbhz"
May 27 21:06:37.017267 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:37.017247    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cwbhz"
May 27 21:06:37.033744 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:37.033686    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pprds" podStartSLOduration=3.139942937 podStartE2EDuration="23.033668996s" podCreationTimestamp="2026-05-27 21:06:14 +0000 UTC" firstStartedPulling="2026-05-27 21:06:16.591819174 +0000 UTC m=+3.230444325" lastFinishedPulling="2026-05-27 21:06:36.485545226 +0000 UTC m=+23.124170384" observedRunningTime="2026-05-27 21:06:37.032824978 +0000 UTC m=+23.671450142" watchObservedRunningTime="2026-05-27 21:06:37.033668996 +0000 UTC m=+23.672294174"
May 27 21:06:37.045736 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:37.045682    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gfsbq" podStartSLOduration=5.124848282 podStartE2EDuration="23.045668148s" podCreationTimestamp="2026-05-27 21:06:14 +0000 UTC" firstStartedPulling="2026-05-27 21:06:16.588154974 +0000 UTC m=+3.226780125" lastFinishedPulling="2026-05-27 21:06:34.508974844 +0000 UTC m=+21.147599991" observedRunningTime="2026-05-27 21:06:37.045219066 +0000 UTC m=+23.683844255" watchObservedRunningTime="2026-05-27 21:06:37.045668148 +0000 UTC m=+23.684293310"
May 27 21:06:37.934435 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:37.934252    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:37.934650 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:37.934252    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:37.934650 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:37.934545    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:37.934650 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:37.934617    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:38.020646 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:38.020608    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r" event={"ID":"98aa4634-5b19-4d18-859e-457982a7dbb5","Type":"ContainerStarted","Data":"67b5efcb4a08741f19d8f8be3d75322345301a47182946cc1677ac23d316991b"}
May 27 21:06:38.930276 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:38.930242    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:38.930491 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:38.930364    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:39.929737 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:39.929703    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:39.930554 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:39.929818    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:39.930554 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:39.929866    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:39.930554 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:39.929955    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:40.027032 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:40.026998    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r" event={"ID":"98aa4634-5b19-4d18-859e-457982a7dbb5","Type":"ContainerStarted","Data":"5437e0101ef6501a045a7d91bc6d1e756f6ace3d0f9486501bae0533adfd6cd5"}
May 27 21:06:40.027264 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:40.027249    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:40.028647 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:40.028622    2570 generic.go:358] "Generic (PLEG): container finished" podID="766a48c1-61da-40cc-be8b-1688536b1722" containerID="0342934128ad1678fe397a10c44114fedb37f7d85ac0cb7b1ca3d62c7ca797b4" exitCode=0
May 27 21:06:40.028757 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:40.028669    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7lpb9" event={"ID":"766a48c1-61da-40cc-be8b-1688536b1722","Type":"ContainerDied","Data":"0342934128ad1678fe397a10c44114fedb37f7d85ac0cb7b1ca3d62c7ca797b4"}
May 27 21:06:40.044213 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:40.044177    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:40.061282 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:40.061230    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r" podStartSLOduration=8.050947183 podStartE2EDuration="26.061214551s" podCreationTimestamp="2026-05-27 21:06:14 +0000 UTC" firstStartedPulling="2026-05-27 21:06:16.588098747 +0000 UTC m=+3.226723892" lastFinishedPulling="2026-05-27 21:06:34.598366116 +0000 UTC m=+21.236991260" observedRunningTime="2026-05-27 21:06:40.060897653 +0000 UTC m=+26.699522818" watchObservedRunningTime="2026-05-27 21:06:40.061214551 +0000 UTC m=+26.699839716"
May 27 21:06:40.930705 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:40.930395    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:40.930705 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:40.930548    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:41.031051 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:41.031022    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:41.031051 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:41.031057    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:41.045870 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:41.045846    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:06:41.640413 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:41.640373    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5ws56"]
May 27 21:06:41.640641 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:41.640501    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:41.640641 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:41.640602    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:41.645996 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:41.645964    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nbt2c"]
May 27 21:06:41.646462 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:41.646440    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:41.646666 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:41.646643    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:41.647907 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:41.647888    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zvqrs"]
May 27 21:06:41.647994 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:41.647983    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:41.648081 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:41.648061    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:42.034111 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:42.034026    2570 generic.go:358] "Generic (PLEG): container finished" podID="766a48c1-61da-40cc-be8b-1688536b1722" containerID="592a737bde0b43312c11169405fef83bc41ecc75fce4f429e5e515ea8dc64d08" exitCode=0
May 27 21:06:42.034461 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:42.034108    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7lpb9" event={"ID":"766a48c1-61da-40cc-be8b-1688536b1722","Type":"ContainerDied","Data":"592a737bde0b43312c11169405fef83bc41ecc75fce4f429e5e515ea8dc64d08"}
May 27 21:06:42.929843 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:42.929804    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:42.930025 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:42.929927    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:43.931001 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:43.930816    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:43.931458 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:43.930891    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:43.931458 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:43.931068    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:43.931458 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:43.931151    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:44.040333 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:44.040225    2570 generic.go:358] "Generic (PLEG): container finished" podID="766a48c1-61da-40cc-be8b-1688536b1722" containerID="5f1115b5e156b2e29dac06f1563d620279ce2ac17720c2c19acc4feb265e251f" exitCode=0
May 27 21:06:44.040333 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:44.040294    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7lpb9" event={"ID":"766a48c1-61da-40cc-be8b-1688536b1722","Type":"ContainerDied","Data":"5f1115b5e156b2e29dac06f1563d620279ce2ac17720c2c19acc4feb265e251f"}
May 27 21:06:44.930228 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:44.930186    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:44.930443 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:44.930325    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:45.929949 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:45.929915    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:45.930431 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:45.930038    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws56" podUID="0216b723-92f1-4f86-a216-384ab8ea7891"
May 27 21:06:45.930431 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:45.930109    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:45.930431 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:45.930218    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvqrs" podUID="8d1f23cf-0c14-4a46-8314-85bc4b106ab4"
May 27 21:06:46.930008 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:46.929974    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:46.930420 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:46.930107    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:06:47.223269 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.223189    2570 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-93.ec2.internal" event="NodeReady"
May 27 21:06:47.223465 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.223362    2570 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 27 21:06:47.264210 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.264175    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"]
May 27 21:06:47.284346 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.284314    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85c7f9878c-zxgmf"]
May 27 21:06:47.284527 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.284505    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:06:47.287851 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.287821    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
May 27 21:06:47.288555 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.288527    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-wdkv5\""
May 27 21:06:47.288878 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.288858    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
May 27 21:06:47.302191 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.302165    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"]
May 27 21:06:47.302336 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.302266    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.304973 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.304902    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 27 21:06:47.305233 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.305198    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 27 21:06:47.305233 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.305209    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 27 21:06:47.305388 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.305247    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xgcft\""
May 27 21:06:47.321318 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.321276    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"]
May 27 21:06:47.321497 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.321441    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"
May 27 21:06:47.321583 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.321503    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 27 21:06:47.324610 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.324544    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 27 21:06:47.324610 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.324606    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 27 21:06:47.324820 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.324731    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 27 21:06:47.324820 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.324813    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 27 21:06:47.337409 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.337374    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2"]
May 27 21:06:47.337581 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.337526    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.341652 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.340871    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 27 21:06:47.341652 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.340924    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 27 21:06:47.341652 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.340871    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 27 21:06:47.341652 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.341191    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 27 21:06:47.348188 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.348151    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"]
May 27 21:06:47.348299 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.348197    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"]
May 27 21:06:47.348299 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.348211    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85c7f9878c-zxgmf"]
May 27 21:06:47.348299 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.348226    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2"]
May 27 21:06:47.348299 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.348240    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4d9hh"]
May 27 21:06:47.348465 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.348301    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2"
May 27 21:06:47.350932 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.350906    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3c91cead-d228-41c8-be05-0b404e6f71d6-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-wzb2c\" (UID: \"3c91cead-d228-41c8-be05-0b404e6f71d6\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:06:47.351097 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.350969    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-wzb2c\" (UID: \"3c91cead-d228-41c8-be05-0b404e6f71d6\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:06:47.351097 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.351040    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ds5h7\""
May 27 21:06:47.351221 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.351203    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 27 21:06:47.362082 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.362052    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ds85j"]
May 27 21:06:47.362282 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.362261    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:47.364783 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.364760    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 27 21:06:47.364922 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.364809    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 27 21:06:47.364922 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.364760    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r8bdp\""
May 27 21:06:47.374400 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.374373    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"]
May 27 21:06:47.374400 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.374405    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ds85j"]
May 27 21:06:47.374649 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.374515    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4d9hh"]
May 27 21:06:47.374649 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.374525    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:06:47.377064 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.376995    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 27 21:06:47.377064 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.377041    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 27 21:06:47.377259 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.376998    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p6pfz\""
May 27 21:06:47.377316 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.377248    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 27 21:06:47.451582 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.451527    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-wzb2c\" (UID: \"3c91cead-d228-41c8-be05-0b404e6f71d6\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:06:47.451582 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.451587    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3ea42c8f-88df-47ae-b348-8f52da40962a-hub\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.451788 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.451614    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg4st\" (UniqueName: \"kubernetes.io/projected/3ea42c8f-88df-47ae-b348-8f52da40962a-kube-api-access-dg4st\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.451788 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.451675    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2hr8\" (UniqueName: \"kubernetes.io/projected/377fa01a-e9e9-49a6-8640-ee79bd8c7d51-kube-api-access-k2hr8\") pod \"klusterlet-addon-workmgr-84547454c5-mjnnr\" (UID: \"377fa01a-e9e9-49a6-8640-ee79bd8c7d51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"
May 27 21:06:47.451788 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.451685    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 27 21:06:47.451788 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.451702    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.451788 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.451742    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b31b724-5935-4809-b314-ce8f4809c8aa-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-574474d94d-wcdn2\" (UID: \"7b31b724-5935-4809-b314-ce8f4809c8aa\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2"
May 27 21:06:47.451788 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.451756    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert podName:3c91cead-d228-41c8-be05-0b404e6f71d6 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:47.951737325 +0000 UTC m=+34.590362467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-wzb2c" (UID: "3c91cead-d228-41c8-be05-0b404e6f71d6") : secret "networking-console-plugin-cert" not found
May 27 21:06:47.452093 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.451790    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94hrq\" (UniqueName: \"kubernetes.io/projected/7b31b724-5935-4809-b314-ce8f4809c8aa-kube-api-access-94hrq\") pod \"managed-serviceaccount-addon-agent-574474d94d-wcdn2\" (UID: \"7b31b724-5935-4809-b314-ce8f4809c8aa\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2"
May 27 21:06:47.452093 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.451837    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3c91cead-d228-41c8-be05-0b404e6f71d6-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-wzb2c\" (UID: \"3c91cead-d228-41c8-be05-0b404e6f71d6\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:06:47.452093 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.451870    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-ca-trust-extracted\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.452093 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.451896    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-trusted-ca\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.452093 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.451922    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41d38e9c-e685-40bc-a1fc-74d393746458-config-volume\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:47.452093 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.451944    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d38e9c-e685-40bc-a1fc-74d393746458-tmp-dir\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:47.452093 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.451976    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msr9d\" (UniqueName: \"kubernetes.io/projected/162181ca-487b-47a0-b0e4-911370859701-kube-api-access-msr9d\") pod \"ingress-canary-ds85j\" (UID: \"162181ca-487b-47a0-b0e4-911370859701\") " pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:06:47.452093 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.452083    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3ea42c8f-88df-47ae-b348-8f52da40962a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.452425 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.452103    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/377fa01a-e9e9-49a6-8640-ee79bd8c7d51-tmp\") pod \"klusterlet-addon-workmgr-84547454c5-mjnnr\" (UID: \"377fa01a-e9e9-49a6-8640-ee79bd8c7d51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"
May 27 21:06:47.452425 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.452122    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert\") pod \"ingress-canary-ds85j\" (UID: \"162181ca-487b-47a0-b0e4-911370859701\") " pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:06:47.452425 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.452136    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-bound-sa-token\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.452425 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.452169    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp2tw\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-kube-api-access-tp2tw\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.452425 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.452214    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3ea42c8f-88df-47ae-b348-8f52da40962a-ca\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.452425 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.452238    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-installation-pull-secrets\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.452425 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.452278    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-certificates\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.452425 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.452327    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ea42c8f-88df-47ae-b348-8f52da40962a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.452425 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.452356    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx8f4\" (UniqueName: \"kubernetes.io/projected/41d38e9c-e685-40bc-a1fc-74d393746458-kube-api-access-rx8f4\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:47.452425 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.452383    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/377fa01a-e9e9-49a6-8640-ee79bd8c7d51-klusterlet-config\") pod \"klusterlet-addon-workmgr-84547454c5-mjnnr\" (UID: \"377fa01a-e9e9-49a6-8640-ee79bd8c7d51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"
May 27 21:06:47.452768 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.452460    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/3ea42c8f-88df-47ae-b348-8f52da40962a-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.452768 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.452492    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:47.452768 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.452542    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-image-registry-private-configuration\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.452768 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.452695    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3c91cead-d228-41c8-be05-0b404e6f71d6-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-wzb2c\" (UID: \"3c91cead-d228-41c8-be05-0b404e6f71d6\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:06:47.553133 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.553033    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-image-registry-private-configuration\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.553133 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.553090    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3ea42c8f-88df-47ae-b348-8f52da40962a-hub\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.553133 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.553115    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dg4st\" (UniqueName: \"kubernetes.io/projected/3ea42c8f-88df-47ae-b348-8f52da40962a-kube-api-access-dg4st\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.553464 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.553156    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hr8\" (UniqueName: \"kubernetes.io/projected/377fa01a-e9e9-49a6-8640-ee79bd8c7d51-kube-api-access-k2hr8\") pod \"klusterlet-addon-workmgr-84547454c5-mjnnr\" (UID: \"377fa01a-e9e9-49a6-8640-ee79bd8c7d51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"
May 27 21:06:47.553464 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.553183    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.553464 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.553221    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b31b724-5935-4809-b314-ce8f4809c8aa-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-574474d94d-wcdn2\" (UID: \"7b31b724-5935-4809-b314-ce8f4809c8aa\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2"
May 27 21:06:47.553464 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.553427    2570 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 21:06:47.553464 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.553449    2570 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85c7f9878c-zxgmf: secret "image-registry-tls" not found
May 27 21:06:47.553728 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.553491    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-94hrq\" (UniqueName: \"kubernetes.io/projected/7b31b724-5935-4809-b314-ce8f4809c8aa-kube-api-access-94hrq\") pod \"managed-serviceaccount-addon-agent-574474d94d-wcdn2\" (UID: \"7b31b724-5935-4809-b314-ce8f4809c8aa\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2"
May 27 21:06:47.553728 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.553505    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls podName:6defcd2e-1e8b-4f5f-abe8-16f4cea4af40 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:48.053486976 +0000 UTC m=+34.692112126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls") pod "image-registry-85c7f9878c-zxgmf" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40") : secret "image-registry-tls" not found
May 27 21:06:47.553728 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.553552    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-ca-trust-extracted\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.553728 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.553598    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-trusted-ca\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.553728 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.553625    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41d38e9c-e685-40bc-a1fc-74d393746458-config-volume\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:47.553728 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.553650    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d38e9c-e685-40bc-a1fc-74d393746458-tmp-dir\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:47.553728 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.553688    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-msr9d\" (UniqueName: \"kubernetes.io/projected/162181ca-487b-47a0-b0e4-911370859701-kube-api-access-msr9d\") pod \"ingress-canary-ds85j\" (UID: \"162181ca-487b-47a0-b0e4-911370859701\") " pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:06:47.553728 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.553713    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3ea42c8f-88df-47ae-b348-8f52da40962a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.554078 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.553997    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d38e9c-e685-40bc-a1fc-74d393746458-tmp-dir\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:47.554078 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.554057    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/377fa01a-e9e9-49a6-8640-ee79bd8c7d51-tmp\") pod \"klusterlet-addon-workmgr-84547454c5-mjnnr\" (UID: \"377fa01a-e9e9-49a6-8640-ee79bd8c7d51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"
May 27 21:06:47.554161 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.554094    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert\") pod \"ingress-canary-ds85j\" (UID: \"162181ca-487b-47a0-b0e4-911370859701\") " pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:06:47.554161 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.554121    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-bound-sa-token\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.554240 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.554162    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tp2tw\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-kube-api-access-tp2tw\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.554240 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.554198    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3ea42c8f-88df-47ae-b348-8f52da40962a-ca\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.554240 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.554233    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-installation-pull-secrets\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.554367 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.554261    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-certificates\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.554367 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.554291    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ea42c8f-88df-47ae-b348-8f52da40962a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.554367 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.554312    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rx8f4\" (UniqueName: \"kubernetes.io/projected/41d38e9c-e685-40bc-a1fc-74d393746458-kube-api-access-rx8f4\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:47.554367 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.554334    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/377fa01a-e9e9-49a6-8640-ee79bd8c7d51-klusterlet-config\") pod \"klusterlet-addon-workmgr-84547454c5-mjnnr\" (UID: \"377fa01a-e9e9-49a6-8640-ee79bd8c7d51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"
May 27 21:06:47.554522 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.554368    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/3ea42c8f-88df-47ae-b348-8f52da40962a-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.554522 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.554397    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:47.554623 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.554526    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 21:06:47.554623 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.554594    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls podName:41d38e9c-e685-40bc-a1fc-74d393746458 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:48.054577115 +0000 UTC m=+34.693202277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls") pod "dns-default-4d9hh" (UID: "41d38e9c-e685-40bc-a1fc-74d393746458") : secret "dns-default-metrics-tls" not found
May 27 21:06:47.554707 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.554668    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-trusted-ca\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.555199 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.555021    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/377fa01a-e9e9-49a6-8640-ee79bd8c7d51-tmp\") pod \"klusterlet-addon-workmgr-84547454c5-mjnnr\" (UID: \"377fa01a-e9e9-49a6-8640-ee79bd8c7d51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"
May 27 21:06:47.555199 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.555107    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 21:06:47.555199 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.555151    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert podName:162181ca-487b-47a0-b0e4-911370859701 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:48.055136345 +0000 UTC m=+34.693761493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert") pod "ingress-canary-ds85j" (UID: "162181ca-487b-47a0-b0e4-911370859701") : secret "canary-serving-cert" not found
May 27 21:06:47.555450 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.555427    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-ca-trust-extracted\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.556019 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.555631    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-certificates\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.556420 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.556389    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/3ea42c8f-88df-47ae-b348-8f52da40962a-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.556515 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.555106    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41d38e9c-e685-40bc-a1fc-74d393746458-config-volume\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:47.558529 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.558501    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-image-registry-private-configuration\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.558658 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.558553    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3ea42c8f-88df-47ae-b348-8f52da40962a-hub\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.558658 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.558620    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b31b724-5935-4809-b314-ce8f4809c8aa-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-574474d94d-wcdn2\" (UID: \"7b31b724-5935-4809-b314-ce8f4809c8aa\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2"
May 27 21:06:47.559313 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.559271    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/377fa01a-e9e9-49a6-8640-ee79bd8c7d51-klusterlet-config\") pod \"klusterlet-addon-workmgr-84547454c5-mjnnr\" (UID: \"377fa01a-e9e9-49a6-8640-ee79bd8c7d51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"
May 27 21:06:47.559653 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.559622    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3ea42c8f-88df-47ae-b348-8f52da40962a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.560201 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.560180    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ea42c8f-88df-47ae-b348-8f52da40962a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.560619 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.560547    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-installation-pull-secrets\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.560698 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.560568    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3ea42c8f-88df-47ae-b348-8f52da40962a-ca\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.564174 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.564135    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg4st\" (UniqueName: \"kubernetes.io/projected/3ea42c8f-88df-47ae-b348-8f52da40962a-kube-api-access-dg4st\") pod \"cluster-proxy-proxy-agent-8466c6768b-gfjhz\" (UID: \"3ea42c8f-88df-47ae-b348-8f52da40962a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.564315 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.564217    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94hrq\" (UniqueName: \"kubernetes.io/projected/7b31b724-5935-4809-b314-ce8f4809c8aa-kube-api-access-94hrq\") pod \"managed-serviceaccount-addon-agent-574474d94d-wcdn2\" (UID: \"7b31b724-5935-4809-b314-ce8f4809c8aa\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2"
May 27 21:06:47.564917 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.564870    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msr9d\" (UniqueName: \"kubernetes.io/projected/162181ca-487b-47a0-b0e4-911370859701-kube-api-access-msr9d\") pod \"ingress-canary-ds85j\" (UID: \"162181ca-487b-47a0-b0e4-911370859701\") " pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:06:47.564917 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.564906    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hr8\" (UniqueName: \"kubernetes.io/projected/377fa01a-e9e9-49a6-8640-ee79bd8c7d51-kube-api-access-k2hr8\") pod \"klusterlet-addon-workmgr-84547454c5-mjnnr\" (UID: \"377fa01a-e9e9-49a6-8640-ee79bd8c7d51\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"
May 27 21:06:47.565089 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.565013    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp2tw\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-kube-api-access-tp2tw\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.565785 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.565763    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx8f4\" (UniqueName: \"kubernetes.io/projected/41d38e9c-e685-40bc-a1fc-74d393746458-kube-api-access-rx8f4\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:47.566812 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.566792    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-bound-sa-token\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:47.633685 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.633647    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"
May 27 21:06:47.655771 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.655733    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs\") pod \"network-metrics-daemon-nbt2c\" (UID: \"375b9e24-ceeb-431a-9102-ca5ac59437cb\") " pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:47.655958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.655830    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgjgg\" (UniqueName: \"kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg\") pod \"network-check-target-zvqrs\" (UID: \"8d1f23cf-0c14-4a46-8314-85bc4b106ab4\") " pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:47.655958 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.655882    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 21:06:47.656071 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.655970    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs podName:375b9e24-ceeb-431a-9102-ca5ac59437cb nodeName:}" failed. No retries permitted until 2026-05-27 21:07:19.655948081 +0000 UTC m=+66.294573229 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs") pod "network-metrics-daemon-nbt2c" (UID: "375b9e24-ceeb-431a-9102-ca5ac59437cb") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 27 21:06:47.656071 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.656014    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 27 21:06:47.656071 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.656033    2570 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 27 21:06:47.656071 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.656042    2570 projected.go:194] Error preparing data for projected volume kube-api-access-jgjgg for pod openshift-network-diagnostics/network-check-target-zvqrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 21:06:47.656258 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.656084    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg podName:8d1f23cf-0c14-4a46-8314-85bc4b106ab4 nodeName:}" failed. No retries permitted until 2026-05-27 21:07:19.656072097 +0000 UTC m=+66.294697247 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgjgg" (UniqueName: "kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg") pod "network-check-target-zvqrs" (UID: "8d1f23cf-0c14-4a46-8314-85bc4b106ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 27 21:06:47.661784 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.661750    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:06:47.671674 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.671637    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2"
May 27 21:06:47.765586 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.763050    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:47.765586 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.763245    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 27 21:06:47.765586 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.763312    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret podName:0216b723-92f1-4f86-a216-384ab8ea7891 nodeName:}" failed. No retries permitted until 2026-05-27 21:07:19.763293781 +0000 UTC m=+66.401918939 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret") pod "global-pull-secret-syncer-5ws56" (UID: "0216b723-92f1-4f86-a216-384ab8ea7891") : object "kube-system"/"original-pull-secret" not registered
May 27 21:06:47.855747 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.855710    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2"]
May 27 21:06:47.858601 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.858412    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"]
May 27 21:06:47.863107 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:47.863077    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377fa01a_e9e9_49a6_8640_ee79bd8c7d51.slice/crio-4c732e86e1e9af0ea2a5c7469765f45b45aebfd0513beae2c38575c309d806e7 WatchSource:0}: Error finding container 4c732e86e1e9af0ea2a5c7469765f45b45aebfd0513beae2c38575c309d806e7: Status 404 returned error can't find the container with id 4c732e86e1e9af0ea2a5c7469765f45b45aebfd0513beae2c38575c309d806e7
May 27 21:06:47.863796 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:47.863644    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b31b724_5935_4809_b314_ce8f4809c8aa.slice/crio-b1cffdd85df9d9ad6045c1bbe4b9949c635724348e809029090405fe9a971a0d WatchSource:0}: Error finding container b1cffdd85df9d9ad6045c1bbe4b9949c635724348e809029090405fe9a971a0d: Status 404 returned error can't find the container with id b1cffdd85df9d9ad6045c1bbe4b9949c635724348e809029090405fe9a971a0d
May 27 21:06:47.870716 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.870687    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"]
May 27 21:06:47.875850 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:06:47.875817    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ea42c8f_88df_47ae_b348_8f52da40962a.slice/crio-b3dcaa03f4e8eb94ab6f8775b9efd79c102407087e407813a23df10e6a3a07fd WatchSource:0}: Error finding container b3dcaa03f4e8eb94ab6f8775b9efd79c102407087e407813a23df10e6a3a07fd: Status 404 returned error can't find the container with id b3dcaa03f4e8eb94ab6f8775b9efd79c102407087e407813a23df10e6a3a07fd
May 27 21:06:47.930132 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.930090    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:06:47.931083 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.930109    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:06:47.933493 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.933264    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 27 21:06:47.933668 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.933283    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gx2x6\""
May 27 21:06:47.933668 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.933309    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 27 21:06:47.933668 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.933311    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 27 21:06:47.964810 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:47.964766    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-wzb2c\" (UID: \"3c91cead-d228-41c8-be05-0b404e6f71d6\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:06:47.964993 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.964945    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 27 21:06:47.965048 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:47.965026    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert podName:3c91cead-d228-41c8-be05-0b404e6f71d6 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:48.965005124 +0000 UTC m=+35.603630281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-wzb2c" (UID: "3c91cead-d228-41c8-be05-0b404e6f71d6") : secret "networking-console-plugin-cert" not found
May 27 21:06:48.049376 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:48.049335    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr" event={"ID":"377fa01a-e9e9-49a6-8640-ee79bd8c7d51","Type":"ContainerStarted","Data":"4c732e86e1e9af0ea2a5c7469765f45b45aebfd0513beae2c38575c309d806e7"}
May 27 21:06:48.050528 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:48.050484    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2" event={"ID":"7b31b724-5935-4809-b314-ce8f4809c8aa","Type":"ContainerStarted","Data":"b1cffdd85df9d9ad6045c1bbe4b9949c635724348e809029090405fe9a971a0d"}
May 27 21:06:48.051668 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:48.051637    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz" event={"ID":"3ea42c8f-88df-47ae-b348-8f52da40962a","Type":"ContainerStarted","Data":"b3dcaa03f4e8eb94ab6f8775b9efd79c102407087e407813a23df10e6a3a07fd"}
May 27 21:06:48.066125 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:48.066040    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert\") pod \"ingress-canary-ds85j\" (UID: \"162181ca-487b-47a0-b0e4-911370859701\") " pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:06:48.066125 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:48.066106    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:48.066351 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:48.066177    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:48.066351 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:48.066202    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 21:06:48.066351 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:48.066280    2570 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 21:06:48.066351 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:48.066295    2570 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85c7f9878c-zxgmf: secret "image-registry-tls" not found
May 27 21:06:48.066351 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:48.066282    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert podName:162181ca-487b-47a0-b0e4-911370859701 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:49.066258075 +0000 UTC m=+35.704883218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert") pod "ingress-canary-ds85j" (UID: "162181ca-487b-47a0-b0e4-911370859701") : secret "canary-serving-cert" not found
May 27 21:06:48.066351 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:48.066279    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 21:06:48.066595 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:48.066357    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls podName:6defcd2e-1e8b-4f5f-abe8-16f4cea4af40 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:49.06634136 +0000 UTC m=+35.704966507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls") pod "image-registry-85c7f9878c-zxgmf" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40") : secret "image-registry-tls" not found
May 27 21:06:48.066595 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:48.066397    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls podName:41d38e9c-e685-40bc-a1fc-74d393746458 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:49.066382101 +0000 UTC m=+35.705007257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls") pod "dns-default-4d9hh" (UID: "41d38e9c-e685-40bc-a1fc-74d393746458") : secret "dns-default-metrics-tls" not found
May 27 21:06:48.930589 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:48.929877    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:06:48.935428 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:48.934928    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 27 21:06:48.935428 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:48.934961    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nvnhn\""
May 27 21:06:48.974158 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:48.974024    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-wzb2c\" (UID: \"3c91cead-d228-41c8-be05-0b404e6f71d6\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:06:48.974352 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:48.974200    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 27 21:06:48.974352 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:48.974283    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert podName:3c91cead-d228-41c8-be05-0b404e6f71d6 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:50.974262379 +0000 UTC m=+37.612887523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-wzb2c" (UID: "3c91cead-d228-41c8-be05-0b404e6f71d6") : secret "networking-console-plugin-cert" not found
May 27 21:06:49.075665 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:49.075137    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:49.075665 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:49.075316    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert\") pod \"ingress-canary-ds85j\" (UID: \"162181ca-487b-47a0-b0e4-911370859701\") " pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:06:49.075665 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:49.075326    2570 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 21:06:49.075665 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:49.075350    2570 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85c7f9878c-zxgmf: secret "image-registry-tls" not found
May 27 21:06:49.075665 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:49.075418    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls podName:6defcd2e-1e8b-4f5f-abe8-16f4cea4af40 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:51.075394916 +0000 UTC m=+37.714020062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls") pod "image-registry-85c7f9878c-zxgmf" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40") : secret "image-registry-tls" not found
May 27 21:06:49.075665 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:49.075401    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:49.075665 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:49.075528    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 21:06:49.075665 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:49.075571    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 21:06:49.075665 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:49.075609    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert podName:162181ca-487b-47a0-b0e4-911370859701 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:51.075592111 +0000 UTC m=+37.714217267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert") pod "ingress-canary-ds85j" (UID: "162181ca-487b-47a0-b0e4-911370859701") : secret "canary-serving-cert" not found
May 27 21:06:49.075665 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:49.075643    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls podName:41d38e9c-e685-40bc-a1fc-74d393746458 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:51.075618196 +0000 UTC m=+37.714243354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls") pod "dns-default-4d9hh" (UID: "41d38e9c-e685-40bc-a1fc-74d393746458") : secret "dns-default-metrics-tls" not found
May 27 21:06:50.994479 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:50.994430    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-wzb2c\" (UID: \"3c91cead-d228-41c8-be05-0b404e6f71d6\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:06:50.995118 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:50.994627    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 27 21:06:50.995118 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:50.994708    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert podName:3c91cead-d228-41c8-be05-0b404e6f71d6 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:54.994686576 +0000 UTC m=+41.633311728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-wzb2c" (UID: "3c91cead-d228-41c8-be05-0b404e6f71d6") : secret "networking-console-plugin-cert" not found
May 27 21:06:51.095371 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:51.095335    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert\") pod \"ingress-canary-ds85j\" (UID: \"162181ca-487b-47a0-b0e4-911370859701\") " pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:06:51.095552 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:51.095397    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:51.095552 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:51.095482    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 21:06:51.095552 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:51.095501    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 21:06:51.095552 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:51.095577    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls podName:41d38e9c-e685-40bc-a1fc-74d393746458 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:55.095536612 +0000 UTC m=+41.734161754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls") pod "dns-default-4d9hh" (UID: "41d38e9c-e685-40bc-a1fc-74d393746458") : secret "dns-default-metrics-tls" not found
May 27 21:06:51.095815 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:51.095602    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert podName:162181ca-487b-47a0-b0e4-911370859701 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:55.095591674 +0000 UTC m=+41.734216817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert") pod "ingress-canary-ds85j" (UID: "162181ca-487b-47a0-b0e4-911370859701") : secret "canary-serving-cert" not found
May 27 21:06:51.095815 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:51.095671    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:51.095815 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:51.095780    2570 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 21:06:51.095815 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:51.095794    2570 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85c7f9878c-zxgmf: secret "image-registry-tls" not found
May 27 21:06:51.095958 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:51.095839    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls podName:6defcd2e-1e8b-4f5f-abe8-16f4cea4af40 nodeName:}" failed. No retries permitted until 2026-05-27 21:06:55.095828055 +0000 UTC m=+41.734453212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls") pod "image-registry-85c7f9878c-zxgmf" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40") : secret "image-registry-tls" not found
May 27 21:06:55.031990 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:55.031947    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-wzb2c\" (UID: \"3c91cead-d228-41c8-be05-0b404e6f71d6\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:06:55.032417 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:55.032099    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 27 21:06:55.032417 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:55.032163    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert podName:3c91cead-d228-41c8-be05-0b404e6f71d6 nodeName:}" failed. No retries permitted until 2026-05-27 21:07:03.032146941 +0000 UTC m=+49.670772090 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-wzb2c" (UID: "3c91cead-d228-41c8-be05-0b404e6f71d6") : secret "networking-console-plugin-cert" not found
May 27 21:06:55.133095 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:55.133052    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:06:55.133271 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:55.133145    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert\") pod \"ingress-canary-ds85j\" (UID: \"162181ca-487b-47a0-b0e4-911370859701\") " pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:06:55.133271 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:55.133198    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:06:55.133271 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:55.133240    2570 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 21:06:55.133271 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:55.133265    2570 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85c7f9878c-zxgmf: secret "image-registry-tls" not found
May 27 21:06:55.133458 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:55.133305    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 21:06:55.133458 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:55.133308    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 21:06:55.133458 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:55.133333    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls podName:6defcd2e-1e8b-4f5f-abe8-16f4cea4af40 nodeName:}" failed. No retries permitted until 2026-05-27 21:07:03.133312031 +0000 UTC m=+49.771937178 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls") pod "image-registry-85c7f9878c-zxgmf" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40") : secret "image-registry-tls" not found
May 27 21:06:55.133458 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:55.133353    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert podName:162181ca-487b-47a0-b0e4-911370859701 nodeName:}" failed. No retries permitted until 2026-05-27 21:07:03.133341009 +0000 UTC m=+49.771966151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert") pod "ingress-canary-ds85j" (UID: "162181ca-487b-47a0-b0e4-911370859701") : secret "canary-serving-cert" not found
May 27 21:06:55.133458 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:06:55.133373    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls podName:41d38e9c-e685-40bc-a1fc-74d393746458 nodeName:}" failed. No retries permitted until 2026-05-27 21:07:03.13336337 +0000 UTC m=+49.771988517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls") pod "dns-default-4d9hh" (UID: "41d38e9c-e685-40bc-a1fc-74d393746458") : secret "dns-default-metrics-tls" not found
May 27 21:06:57.074508 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:57.074471    2570 generic.go:358] "Generic (PLEG): container finished" podID="766a48c1-61da-40cc-be8b-1688536b1722" containerID="f4b2e58a173073092f61a1374111c5413f15e962c3f7be13ab1c73f2ff24fa41" exitCode=0
May 27 21:06:57.074986 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:57.074537    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7lpb9" event={"ID":"766a48c1-61da-40cc-be8b-1688536b1722","Type":"ContainerDied","Data":"f4b2e58a173073092f61a1374111c5413f15e962c3f7be13ab1c73f2ff24fa41"}
May 27 21:06:57.075861 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:57.075830    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2" event={"ID":"7b31b724-5935-4809-b314-ce8f4809c8aa","Type":"ContainerStarted","Data":"1b8fb447216a09ecda5186936f1bbbd1e1c501f4e9637269f0b8326bcb34a1e7"}
May 27 21:06:57.077100 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:57.077078    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz" event={"ID":"3ea42c8f-88df-47ae-b348-8f52da40962a","Type":"ContainerStarted","Data":"3536ba52316f9ace9be8b2c453ed9ce335fddda5de64c740f7d7ffaeee03f20d"}
May 27 21:06:57.078362 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:57.078335    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr" event={"ID":"377fa01a-e9e9-49a6-8640-ee79bd8c7d51","Type":"ContainerStarted","Data":"19764fe11251e5499f127d0232ce685eec7bae09aaf656fea241796c2d7de326"}
May 27 21:06:57.078582 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:57.078548    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"
May 27 21:06:57.080165 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:57.080147    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"
May 27 21:06:57.118085 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:57.118021    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr" podStartSLOduration=6.090820186 podStartE2EDuration="15.117990054s" podCreationTimestamp="2026-05-27 21:06:42 +0000 UTC" firstStartedPulling="2026-05-27 21:06:47.865291779 +0000 UTC m=+34.503916928" lastFinishedPulling="2026-05-27 21:06:56.892461653 +0000 UTC m=+43.531086796" observedRunningTime="2026-05-27 21:06:57.117422143 +0000 UTC m=+43.756047324" watchObservedRunningTime="2026-05-27 21:06:57.117990054 +0000 UTC m=+43.756615217"
May 27 21:06:57.133085 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:57.133036    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2" podStartSLOduration=6.366714824 podStartE2EDuration="15.133024559s" podCreationTimestamp="2026-05-27 21:06:42 +0000 UTC" firstStartedPulling="2026-05-27 21:06:47.865787262 +0000 UTC m=+34.504412412" lastFinishedPulling="2026-05-27 21:06:56.632096992 +0000 UTC m=+43.270722147" observedRunningTime="2026-05-27 21:06:57.132271053 +0000 UTC m=+43.770896220" watchObservedRunningTime="2026-05-27 21:06:57.133024559 +0000 UTC m=+43.771649724"
May 27 21:06:58.083295 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:58.083256    2570 generic.go:358] "Generic (PLEG): container finished" podID="766a48c1-61da-40cc-be8b-1688536b1722" containerID="c0ccc0563bd0660a8673932bb2865d1cd47e46cd96da6d0a834b33b5375552e3" exitCode=0
May 27 21:06:58.083855 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:58.083335    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7lpb9" event={"ID":"766a48c1-61da-40cc-be8b-1688536b1722","Type":"ContainerDied","Data":"c0ccc0563bd0660a8673932bb2865d1cd47e46cd96da6d0a834b33b5375552e3"}
May 27 21:06:59.087943 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:59.087858    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7lpb9" event={"ID":"766a48c1-61da-40cc-be8b-1688536b1722","Type":"ContainerStarted","Data":"735aa2011f1fc1ec3a242ebd7d8effa32896eba7d2b0a897af9866cff95c8335"}
May 27 21:06:59.089609 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:59.089582    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz" event={"ID":"3ea42c8f-88df-47ae-b348-8f52da40962a","Type":"ContainerStarted","Data":"fdec68b137abfa12d34342a0fd764945934eb53e80cdf87cc2663277381ca2a1"}
May 27 21:06:59.089609 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:59.089612    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz" event={"ID":"3ea42c8f-88df-47ae-b348-8f52da40962a","Type":"ContainerStarted","Data":"63c67e3d850869ddc282620a3699c007b210f5948cd7b11a41b2e15c900fb260"}
May 27 21:06:59.113455 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:59.113404    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7lpb9" podStartSLOduration=5.04668612 podStartE2EDuration="45.113388493s" podCreationTimestamp="2026-05-27 21:06:14 +0000 UTC" firstStartedPulling="2026-05-27 21:06:16.566257627 +0000 UTC m=+3.204882776" lastFinishedPulling="2026-05-27 21:06:56.63295999 +0000 UTC m=+43.271585149" observedRunningTime="2026-05-27 21:06:59.112685084 +0000 UTC m=+45.751310248" watchObservedRunningTime="2026-05-27 21:06:59.113388493 +0000 UTC m=+45.752013659"
May 27 21:06:59.131201 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:06:59.131153    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz" podStartSLOduration=6.249230563 podStartE2EDuration="17.131141198s" podCreationTimestamp="2026-05-27 21:06:42 +0000 UTC" firstStartedPulling="2026-05-27 21:06:47.877948096 +0000 UTC m=+34.516573241" lastFinishedPulling="2026-05-27 21:06:58.759858719 +0000 UTC m=+45.398483876" observedRunningTime="2026-05-27 21:06:59.130302092 +0000 UTC m=+45.768927283" watchObservedRunningTime="2026-05-27 21:06:59.131141198 +0000 UTC m=+45.769766398"
May 27 21:07:03.103982 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:03.103937    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-wzb2c\" (UID: \"3c91cead-d228-41c8-be05-0b404e6f71d6\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:07:03.104373 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:03.104075    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 27 21:07:03.104373 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:03.104137    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert podName:3c91cead-d228-41c8-be05-0b404e6f71d6 nodeName:}" failed. No retries permitted until 2026-05-27 21:07:19.10412062 +0000 UTC m=+65.742745763 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-wzb2c" (UID: "3c91cead-d228-41c8-be05-0b404e6f71d6") : secret "networking-console-plugin-cert" not found
May 27 21:07:03.204313 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:03.204267    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:07:03.204511 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:03.204333    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert\") pod \"ingress-canary-ds85j\" (UID: \"162181ca-487b-47a0-b0e4-911370859701\") " pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:07:03.204511 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:03.204366    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:07:03.204511 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:03.204399    2570 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 21:07:03.204511 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:03.204424    2570 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85c7f9878c-zxgmf: secret "image-registry-tls" not found
May 27 21:07:03.204511 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:03.204469    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 21:07:03.204511 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:03.204493    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls podName:6defcd2e-1e8b-4f5f-abe8-16f4cea4af40 nodeName:}" failed. No retries permitted until 2026-05-27 21:07:19.204470353 +0000 UTC m=+65.843095507 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls") pod "image-registry-85c7f9878c-zxgmf" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40") : secret "image-registry-tls" not found
May 27 21:07:03.204511 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:03.204511    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls podName:41d38e9c-e685-40bc-a1fc-74d393746458 nodeName:}" failed. No retries permitted until 2026-05-27 21:07:19.204499782 +0000 UTC m=+65.843124924 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls") pod "dns-default-4d9hh" (UID: "41d38e9c-e685-40bc-a1fc-74d393746458") : secret "dns-default-metrics-tls" not found
May 27 21:07:03.204511 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:03.204468    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 21:07:03.204937 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:03.204540    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert podName:162181ca-487b-47a0-b0e4-911370859701 nodeName:}" failed. No retries permitted until 2026-05-27 21:07:19.20452909 +0000 UTC m=+65.843154234 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert") pod "ingress-canary-ds85j" (UID: "162181ca-487b-47a0-b0e4-911370859701") : secret "canary-serving-cert" not found
May 27 21:07:13.056180 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:13.056150    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mwl2r"
May 27 21:07:19.132929 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:19.132880    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-wzb2c\" (UID: \"3c91cead-d228-41c8-be05-0b404e6f71d6\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:07:19.133321 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:19.133030    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 27 21:07:19.133321 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:19.133102    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert podName:3c91cead-d228-41c8-be05-0b404e6f71d6 nodeName:}" failed. No retries permitted until 2026-05-27 21:07:51.133082884 +0000 UTC m=+97.771708031 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-wzb2c" (UID: "3c91cead-d228-41c8-be05-0b404e6f71d6") : secret "networking-console-plugin-cert" not found
May 27 21:07:19.234230 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:19.234185    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:07:19.234391 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:19.234257    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:07:19.234391 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:19.234292    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert\") pod \"ingress-canary-ds85j\" (UID: \"162181ca-487b-47a0-b0e4-911370859701\") " pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:07:19.234391 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:19.234342    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 21:07:19.234391 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:19.234371    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 21:07:19.234391 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:19.234388    2570 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 21:07:19.234546 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:19.234405    2570 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85c7f9878c-zxgmf: secret "image-registry-tls" not found
May 27 21:07:19.234546 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:19.234421    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls podName:41d38e9c-e685-40bc-a1fc-74d393746458 nodeName:}" failed. No retries permitted until 2026-05-27 21:07:51.234398971 +0000 UTC m=+97.873024114 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls") pod "dns-default-4d9hh" (UID: "41d38e9c-e685-40bc-a1fc-74d393746458") : secret "dns-default-metrics-tls" not found
May 27 21:07:19.234546 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:19.234440    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls podName:6defcd2e-1e8b-4f5f-abe8-16f4cea4af40 nodeName:}" failed. No retries permitted until 2026-05-27 21:07:51.234428094 +0000 UTC m=+97.873053258 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls") pod "image-registry-85c7f9878c-zxgmf" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40") : secret "image-registry-tls" not found
May 27 21:07:19.234546 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:19.234459    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert podName:162181ca-487b-47a0-b0e4-911370859701 nodeName:}" failed. No retries permitted until 2026-05-27 21:07:51.234451351 +0000 UTC m=+97.873076494 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert") pod "ingress-canary-ds85j" (UID: "162181ca-487b-47a0-b0e4-911370859701") : secret "canary-serving-cert" not found
May 27 21:07:19.738342 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:19.738303    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs\") pod \"network-metrics-daemon-nbt2c\" (UID: \"375b9e24-ceeb-431a-9102-ca5ac59437cb\") " pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:07:19.738512 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:19.738374    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jgjgg\" (UniqueName: \"kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg\") pod \"network-check-target-zvqrs\" (UID: \"8d1f23cf-0c14-4a46-8314-85bc4b106ab4\") " pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:07:19.741272 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:19.741252    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 27 21:07:19.741315 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:19.741267    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 27 21:07:19.749494 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:19.749474    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 27 21:07:19.749543 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:19.749534    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs podName:375b9e24-ceeb-431a-9102-ca5ac59437cb nodeName:}" failed. No retries permitted until 2026-05-27 21:08:23.749518499 +0000 UTC m=+130.388143647 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs") pod "network-metrics-daemon-nbt2c" (UID: "375b9e24-ceeb-431a-9102-ca5ac59437cb") : secret "metrics-daemon-secret" not found
May 27 21:07:19.752114 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:19.752092    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 27 21:07:19.763647 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:19.763616    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgjgg\" (UniqueName: \"kubernetes.io/projected/8d1f23cf-0c14-4a46-8314-85bc4b106ab4-kube-api-access-jgjgg\") pod \"network-check-target-zvqrs\" (UID: \"8d1f23cf-0c14-4a46-8314-85bc4b106ab4\") " pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:07:19.839080 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:19.839042    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:07:19.841753 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:19.841732    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 27 21:07:19.852049 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:19.852026    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0216b723-92f1-4f86-a216-384ab8ea7891-original-pull-secret\") pod \"global-pull-secret-syncer-5ws56\" (UID: \"0216b723-92f1-4f86-a216-384ab8ea7891\") " pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:07:20.043985 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:20.043898    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws56"
May 27 21:07:20.053368 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:20.053344    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gx2x6\""
May 27 21:07:20.061577 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:20.061540    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:07:20.180948 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:20.180902    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5ws56"]
May 27 21:07:20.184397 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:07:20.184365    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0216b723_92f1_4f86_a216_384ab8ea7891.slice/crio-c2425d31c10828fc70b103bbb78004f8263981b34c21702ef60559a4dcc27c90 WatchSource:0}: Error finding container c2425d31c10828fc70b103bbb78004f8263981b34c21702ef60559a4dcc27c90: Status 404 returned error can't find the container with id c2425d31c10828fc70b103bbb78004f8263981b34c21702ef60559a4dcc27c90
May 27 21:07:20.196498 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:20.196472    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zvqrs"]
May 27 21:07:20.199810 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:07:20.199784    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1f23cf_0c14_4a46_8314_85bc4b106ab4.slice/crio-ea5fc0900d6b19b9e9eb6cdc2385849cf77f160867fed3076b4c66e831d6801b WatchSource:0}: Error finding container ea5fc0900d6b19b9e9eb6cdc2385849cf77f160867fed3076b4c66e831d6801b: Status 404 returned error can't find the container with id ea5fc0900d6b19b9e9eb6cdc2385849cf77f160867fed3076b4c66e831d6801b
May 27 21:07:21.134077 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:21.134037    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zvqrs" event={"ID":"8d1f23cf-0c14-4a46-8314-85bc4b106ab4","Type":"ContainerStarted","Data":"ea5fc0900d6b19b9e9eb6cdc2385849cf77f160867fed3076b4c66e831d6801b"}
May 27 21:07:21.135167 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:21.135137    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5ws56" event={"ID":"0216b723-92f1-4f86-a216-384ab8ea7891","Type":"ContainerStarted","Data":"c2425d31c10828fc70b103bbb78004f8263981b34c21702ef60559a4dcc27c90"}
May 27 21:07:26.148452 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:26.148414    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zvqrs" event={"ID":"8d1f23cf-0c14-4a46-8314-85bc4b106ab4","Type":"ContainerStarted","Data":"8542c5fb2eac2911068fe7c8fb89cdf89f7911c69a37c1d9bc5fc2c613a1dc51"}
May 27 21:07:26.148994 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:26.148642    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:07:26.151834 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:26.151804    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5ws56" event={"ID":"0216b723-92f1-4f86-a216-384ab8ea7891","Type":"ContainerStarted","Data":"239b2830f3b62c3240bdc58666e5900cf894207c688ba641d34099471366b83b"}
May 27 21:07:26.167436 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:26.167375    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zvqrs" podStartSLOduration=66.597977733 podStartE2EDuration="1m12.167359368s" podCreationTimestamp="2026-05-27 21:06:14 +0000 UTC" firstStartedPulling="2026-05-27 21:07:20.201703911 +0000 UTC m=+66.840329053" lastFinishedPulling="2026-05-27 21:07:25.771085541 +0000 UTC m=+72.409710688" observedRunningTime="2026-05-27 21:07:26.166462173 +0000 UTC m=+72.805087340" watchObservedRunningTime="2026-05-27 21:07:26.167359368 +0000 UTC m=+72.805984532"
May 27 21:07:26.184893 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:26.184843    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5ws56" podStartSLOduration=65.597111749 podStartE2EDuration="1m11.184826104s" podCreationTimestamp="2026-05-27 21:06:15 +0000 UTC" firstStartedPulling="2026-05-27 21:07:20.186057451 +0000 UTC m=+66.824682594" lastFinishedPulling="2026-05-27 21:07:25.773771803 +0000 UTC m=+72.412396949" observedRunningTime="2026-05-27 21:07:26.18404314 +0000 UTC m=+72.822668306" watchObservedRunningTime="2026-05-27 21:07:26.184826104 +0000 UTC m=+72.823451268"
May 27 21:07:51.195120 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:51.195078    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-wzb2c\" (UID: \"3c91cead-d228-41c8-be05-0b404e6f71d6\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:07:51.195682 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:51.195232    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 27 21:07:51.195682 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:51.195315    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert podName:3c91cead-d228-41c8-be05-0b404e6f71d6 nodeName:}" failed. No retries permitted until 2026-05-27 21:08:55.195297207 +0000 UTC m=+161.833922349 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-wzb2c" (UID: "3c91cead-d228-41c8-be05-0b404e6f71d6") : secret "networking-console-plugin-cert" not found
May 27 21:07:51.295598 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:51.295541    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:07:51.295769 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:51.295608    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert\") pod \"ingress-canary-ds85j\" (UID: \"162181ca-487b-47a0-b0e4-911370859701\") " pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:07:51.295769 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:51.295639    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:07:51.295769 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:51.295696    2570 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 27 21:07:51.295769 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:51.295720    2570 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85c7f9878c-zxgmf: secret "image-registry-tls" not found
May 27 21:07:51.295769 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:51.295735    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 27 21:07:51.295769 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:51.295741    2570 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 27 21:07:51.295955 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:51.295785    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls podName:41d38e9c-e685-40bc-a1fc-74d393746458 nodeName:}" failed. No retries permitted until 2026-05-27 21:08:55.295771234 +0000 UTC m=+161.934396377 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls") pod "dns-default-4d9hh" (UID: "41d38e9c-e685-40bc-a1fc-74d393746458") : secret "dns-default-metrics-tls" not found
May 27 21:07:51.295955 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:51.295798    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls podName:6defcd2e-1e8b-4f5f-abe8-16f4cea4af40 nodeName:}" failed. No retries permitted until 2026-05-27 21:08:55.295791794 +0000 UTC m=+161.934416939 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls") pod "image-registry-85c7f9878c-zxgmf" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40") : secret "image-registry-tls" not found
May 27 21:07:51.295955 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:07:51.295813    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert podName:162181ca-487b-47a0-b0e4-911370859701 nodeName:}" failed. No retries permitted until 2026-05-27 21:08:55.295807548 +0000 UTC m=+161.934432691 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert") pod "ingress-canary-ds85j" (UID: "162181ca-487b-47a0-b0e4-911370859701") : secret "canary-serving-cert" not found
May 27 21:07:57.158123 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:07:57.158087    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zvqrs"
May 27 21:08:22.387416 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:22.387386    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7cjkr_a31a703f-7353-451c-9112-accce453fe8c/dns-node-resolver/0.log"
May 27 21:08:23.387314 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:23.387274    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ktm2q_8627c382-6754-44fb-bb08-de5c1cd78407/node-ca/0.log"
May 27 21:08:23.844404 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:23.844309    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs\") pod \"network-metrics-daemon-nbt2c\" (UID: \"375b9e24-ceeb-431a-9102-ca5ac59437cb\") " pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:08:23.844787 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:23.844465    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 27 21:08:23.844787 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:23.844532    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs podName:375b9e24-ceeb-431a-9102-ca5ac59437cb nodeName:}" failed. No retries permitted until 2026-05-27 21:10:25.844514586 +0000 UTC m=+252.483139728 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs") pod "network-metrics-daemon-nbt2c" (UID: "375b9e24-ceeb-431a-9102-ca5ac59437cb") : secret "metrics-daemon-secret" not found
May 27 21:08:34.020813 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.020778    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s2fv5"]
May 27 21:08:34.025387 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.025361    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.028714 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.028688    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 27 21:08:34.028832 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.028714    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 27 21:08:34.028832 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.028715    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-96bzc\""
May 27 21:08:34.030386 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.030365    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 27 21:08:34.030386 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.030381    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 27 21:08:34.041537 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.041511    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s2fv5"]
May 27 21:08:34.126424 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.126387    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.126620 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.126443    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/02a18b4c-800c-461c-ab22-6ec305befca3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.126620 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.126528    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/02a18b4c-800c-461c-ab22-6ec305befca3-data-volume\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.126695 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.126628    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/02a18b4c-800c-461c-ab22-6ec305befca3-crio-socket\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.126695 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.126651    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7rvr\" (UniqueName: \"kubernetes.io/projected/02a18b4c-800c-461c-ab22-6ec305befca3-kube-api-access-p7rvr\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.227708 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.227656    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/02a18b4c-800c-461c-ab22-6ec305befca3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.227895 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.227731    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/02a18b4c-800c-461c-ab22-6ec305befca3-data-volume\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.227895 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.227772    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/02a18b4c-800c-461c-ab22-6ec305befca3-crio-socket\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.227895 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.227787    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p7rvr\" (UniqueName: \"kubernetes.io/projected/02a18b4c-800c-461c-ab22-6ec305befca3-kube-api-access-p7rvr\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.227895 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.227838    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.227895 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.227877    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/02a18b4c-800c-461c-ab22-6ec305befca3-crio-socket\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.228071 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:34.227955    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
May 27 21:08:34.228071 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:34.228015    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls podName:02a18b4c-800c-461c-ab22-6ec305befca3 nodeName:}" failed. No retries permitted until 2026-05-27 21:08:34.728001617 +0000 UTC m=+141.366626760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2fv5" (UID: "02a18b4c-800c-461c-ab22-6ec305befca3") : secret "insights-runtime-extractor-tls" not found
May 27 21:08:34.228147 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.228073    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/02a18b4c-800c-461c-ab22-6ec305befca3-data-volume\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.228299 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.228277    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/02a18b4c-800c-461c-ab22-6ec305befca3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.236853 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.236827    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7rvr\" (UniqueName: \"kubernetes.io/projected/02a18b4c-800c-461c-ab22-6ec305befca3-kube-api-access-p7rvr\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.731719 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:34.731677    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:34.731895 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:34.731832    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
May 27 21:08:34.731938 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:34.731909    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls podName:02a18b4c-800c-461c-ab22-6ec305befca3 nodeName:}" failed. No retries permitted until 2026-05-27 21:08:35.731893103 +0000 UTC m=+142.370518245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2fv5" (UID: "02a18b4c-800c-461c-ab22-6ec305befca3") : secret "insights-runtime-extractor-tls" not found
May 27 21:08:35.738909 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:35.738871    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:35.739310 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:35.739022    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
May 27 21:08:35.739310 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:35.739088    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls podName:02a18b4c-800c-461c-ab22-6ec305befca3 nodeName:}" failed. No retries permitted until 2026-05-27 21:08:37.739072292 +0000 UTC m=+144.377697438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2fv5" (UID: "02a18b4c-800c-461c-ab22-6ec305befca3") : secret "insights-runtime-extractor-tls" not found
May 27 21:08:37.753772 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:37.753728    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:37.754225 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:37.753872    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
May 27 21:08:37.754225 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:37.753940    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls podName:02a18b4c-800c-461c-ab22-6ec305befca3 nodeName:}" failed. No retries permitted until 2026-05-27 21:08:41.753922516 +0000 UTC m=+148.392547658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2fv5" (UID: "02a18b4c-800c-461c-ab22-6ec305befca3") : secret "insights-runtime-extractor-tls" not found
May 27 21:08:41.782840 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:41.782801    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:41.783339 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:41.782947    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
May 27 21:08:41.783339 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:41.783007    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls podName:02a18b4c-800c-461c-ab22-6ec305befca3 nodeName:}" failed. No retries permitted until 2026-05-27 21:08:49.782991162 +0000 UTC m=+156.421616310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2fv5" (UID: "02a18b4c-800c-461c-ab22-6ec305befca3") : secret "insights-runtime-extractor-tls" not found
May 27 21:08:49.851842 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:49.851799    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:49.854043 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:49.854022    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/02a18b4c-800c-461c-ab22-6ec305befca3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2fv5\" (UID: \"02a18b4c-800c-461c-ab22-6ec305befca3\") " pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:49.934064 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:49.934031    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s2fv5"
May 27 21:08:50.055346 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:50.055314    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s2fv5"]
May 27 21:08:50.059289 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:08:50.059263    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a18b4c_800c_461c_ab22_6ec305befca3.slice/crio-c30caf23dedfd6097757b06b4419d82463e9f4e9667e0ab0deb36599e6048699 WatchSource:0}: Error finding container c30caf23dedfd6097757b06b4419d82463e9f4e9667e0ab0deb36599e6048699: Status 404 returned error can't find the container with id c30caf23dedfd6097757b06b4419d82463e9f4e9667e0ab0deb36599e6048699
May 27 21:08:50.298049 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:50.298003    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c" podUID="3c91cead-d228-41c8-be05-0b404e6f71d6"
May 27 21:08:50.315190 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:50.315156    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf" podUID="6defcd2e-1e8b-4f5f-abe8-16f4cea4af40"
May 27 21:08:50.347111 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:50.347082    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:08:50.347111 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:50.347090    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2fv5" event={"ID":"02a18b4c-800c-461c-ab22-6ec305befca3","Type":"ContainerStarted","Data":"72b7b05617fd15acdc705708f2ebecf9d9cd59dc8e6b5a27ddcedcc93e2bda42"}
May 27 21:08:50.347292 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:50.347131    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2fv5" event={"ID":"02a18b4c-800c-461c-ab22-6ec305befca3","Type":"ContainerStarted","Data":"c30caf23dedfd6097757b06b4419d82463e9f4e9667e0ab0deb36599e6048699"}
May 27 21:08:50.347292 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:50.347244    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:08:50.388863 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:50.388825    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4d9hh" podUID="41d38e9c-e685-40bc-a1fc-74d393746458"
May 27 21:08:50.393981 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:50.393956    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-ds85j" podUID="162181ca-487b-47a0-b0e4-911370859701"
May 27 21:08:51.351363 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:51.351325    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:08:51.351820 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:51.351374    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4d9hh"
May 27 21:08:51.351820 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:51.351318    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2fv5" event={"ID":"02a18b4c-800c-461c-ab22-6ec305befca3","Type":"ContainerStarted","Data":"5d20690b13122177366a63ae133ab312c0095027597e67eb08a31b76113d9ac2"}
May 27 21:08:51.944439 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:08:51.944395    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-nbt2c" podUID="375b9e24-ceeb-431a-9102-ca5ac59437cb"
May 27 21:08:52.355294 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:52.355257    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2fv5" event={"ID":"02a18b4c-800c-461c-ab22-6ec305befca3","Type":"ContainerStarted","Data":"2ae66a0620c52c4fb82def60626d087df364cc7661fea8f70c62986a05e435d1"}
May 27 21:08:52.377078 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:52.377023    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s2fv5" podStartSLOduration=16.238028601 podStartE2EDuration="18.377007956s" podCreationTimestamp="2026-05-27 21:08:34 +0000 UTC" firstStartedPulling="2026-05-27 21:08:50.11164526 +0000 UTC m=+156.750270416" lastFinishedPulling="2026-05-27 21:08:52.250624629 +0000 UTC m=+158.889249771" observedRunningTime="2026-05-27 21:08:52.376609123 +0000 UTC m=+159.015234288" watchObservedRunningTime="2026-05-27 21:08:52.377007956 +0000 UTC m=+159.015633158"
May 27 21:08:55.196423 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.196378    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-wzb2c\" (UID: \"3c91cead-d228-41c8-be05-0b404e6f71d6\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:08:55.198791 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.198764    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c91cead-d228-41c8-be05-0b404e6f71d6-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-wzb2c\" (UID: \"3c91cead-d228-41c8-be05-0b404e6f71d6\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:08:55.297062 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.297027    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:08:55.297220 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.297086    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:08:55.297220 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.297126    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert\") pod \"ingress-canary-ds85j\" (UID: \"162181ca-487b-47a0-b0e4-911370859701\") " pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:08:55.299445 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.299419    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d38e9c-e685-40bc-a1fc-74d393746458-metrics-tls\") pod \"dns-default-4d9hh\" (UID: \"41d38e9c-e685-40bc-a1fc-74d393746458\") " pod="openshift-dns/dns-default-4d9hh"
May 27 21:08:55.299595 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.299540    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls\") pod \"image-registry-85c7f9878c-zxgmf\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") " pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:08:55.299731 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.299716    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/162181ca-487b-47a0-b0e4-911370859701-cert\") pod \"ingress-canary-ds85j\" (UID: \"162181ca-487b-47a0-b0e4-911370859701\") " pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:08:55.451785 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.451698    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-wdkv5\""
May 27 21:08:55.451785 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.451712    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xgcft\""
May 27 21:08:55.458646 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.458618    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"
May 27 21:08:55.458742 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.458704    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:08:55.557463 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.556994    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p6pfz\""
May 27 21:08:55.557463 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.557312    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r8bdp\""
May 27 21:08:55.562648 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.562622    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ds85j"
May 27 21:08:55.563576 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.563519    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4d9hh"
May 27 21:08:55.599046 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.598955    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85c7f9878c-zxgmf"]
May 27 21:08:55.603812 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:08:55.603776    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6defcd2e_1e8b_4f5f_abe8_16f4cea4af40.slice/crio-6192fec2e8c02db350a19422917683001894dba36ff0a85b4a218b7fabfdc858 WatchSource:0}: Error finding container 6192fec2e8c02db350a19422917683001894dba36ff0a85b4a218b7fabfdc858: Status 404 returned error can't find the container with id 6192fec2e8c02db350a19422917683001894dba36ff0a85b4a218b7fabfdc858
May 27 21:08:55.620826 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.620785    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c"]
May 27 21:08:55.623447 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:08:55.623412    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c91cead_d228_41c8_be05_0b404e6f71d6.slice/crio-c8ca829c301ed3acd99558f0928492cf57811d4b7b3e53636b3ffe528e0a48f0 WatchSource:0}: Error finding container c8ca829c301ed3acd99558f0928492cf57811d4b7b3e53636b3ffe528e0a48f0: Status 404 returned error can't find the container with id c8ca829c301ed3acd99558f0928492cf57811d4b7b3e53636b3ffe528e0a48f0
May 27 21:08:55.713003 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.712889    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4d9hh"]
May 27 21:08:55.716661 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:08:55.716622    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d38e9c_e685_40bc_a1fc_74d393746458.slice/crio-f43b5b979f7a5b3f05e7e1ad3e3b44b924c4d65804b631a3a01db12fcf7bad84 WatchSource:0}: Error finding container f43b5b979f7a5b3f05e7e1ad3e3b44b924c4d65804b631a3a01db12fcf7bad84: Status 404 returned error can't find the container with id f43b5b979f7a5b3f05e7e1ad3e3b44b924c4d65804b631a3a01db12fcf7bad84
May 27 21:08:55.732382 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:55.732358    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ds85j"]
May 27 21:08:55.735569 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:08:55.735523    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162181ca_487b_47a0_b0e4_911370859701.slice/crio-5a42b7ba9de46bdc7b650251d25d39dd5cc53f8a4fd3292f436d9e9c761783b3 WatchSource:0}: Error finding container 5a42b7ba9de46bdc7b650251d25d39dd5cc53f8a4fd3292f436d9e9c761783b3: Status 404 returned error can't find the container with id 5a42b7ba9de46bdc7b650251d25d39dd5cc53f8a4fd3292f436d9e9c761783b3
May 27 21:08:56.367670 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:56.367451    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4d9hh" event={"ID":"41d38e9c-e685-40bc-a1fc-74d393746458","Type":"ContainerStarted","Data":"f43b5b979f7a5b3f05e7e1ad3e3b44b924c4d65804b631a3a01db12fcf7bad84"}
May 27 21:08:56.371740 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:56.371702    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf" event={"ID":"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40","Type":"ContainerStarted","Data":"31983137d2d61a2308356d9762e7daca266186072b35e0add906f6b0fee36b87"}
May 27 21:08:56.371895 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:56.371752    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf" event={"ID":"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40","Type":"ContainerStarted","Data":"6192fec2e8c02db350a19422917683001894dba36ff0a85b4a218b7fabfdc858"}
May 27 21:08:56.372240 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:56.372220    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:08:56.373760 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:56.373732    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ds85j" event={"ID":"162181ca-487b-47a0-b0e4-911370859701","Type":"ContainerStarted","Data":"5a42b7ba9de46bdc7b650251d25d39dd5cc53f8a4fd3292f436d9e9c761783b3"}
May 27 21:08:56.375136 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:56.375103    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c" event={"ID":"3c91cead-d228-41c8-be05-0b404e6f71d6","Type":"ContainerStarted","Data":"c8ca829c301ed3acd99558f0928492cf57811d4b7b3e53636b3ffe528e0a48f0"}
May 27 21:08:56.396510 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:56.396378    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf" podStartSLOduration=162.396359682 podStartE2EDuration="2m42.396359682s" podCreationTimestamp="2026-05-27 21:06:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 21:08:56.394552698 +0000 UTC m=+163.033177888" watchObservedRunningTime="2026-05-27 21:08:56.396359682 +0000 UTC m=+163.034984847"
May 27 21:08:57.079008 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:57.078961    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr" podUID="377fa01a-e9e9-49a6-8640-ee79bd8c7d51" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.10:8000/readyz\": dial tcp 10.132.0.10:8000: connect: connection refused"
May 27 21:08:57.379995 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:57.379960    2570 generic.go:358] "Generic (PLEG): container finished" podID="7b31b724-5935-4809-b314-ce8f4809c8aa" containerID="1b8fb447216a09ecda5186936f1bbbd1e1c501f4e9637269f0b8326bcb34a1e7" exitCode=255
May 27 21:08:57.380441 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:57.380015    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2" event={"ID":"7b31b724-5935-4809-b314-ce8f4809c8aa","Type":"ContainerDied","Data":"1b8fb447216a09ecda5186936f1bbbd1e1c501f4e9637269f0b8326bcb34a1e7"}
May 27 21:08:57.380441 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:57.380383    2570 scope.go:117] "RemoveContainer" containerID="1b8fb447216a09ecda5186936f1bbbd1e1c501f4e9637269f0b8326bcb34a1e7"
May 27 21:08:57.381682 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:57.381593    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c" event={"ID":"3c91cead-d228-41c8-be05-0b404e6f71d6","Type":"ContainerStarted","Data":"4b9dbc4078f232ce40d35488cfc6021d32d9776704aeb692ada72e9da59438e2"}
May 27 21:08:57.412755 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:57.412706    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wzb2c" podStartSLOduration=156.43145818 podStartE2EDuration="2m37.412691181s" podCreationTimestamp="2026-05-27 21:06:20 +0000 UTC" firstStartedPulling="2026-05-27 21:08:55.626107402 +0000 UTC m=+162.264732544" lastFinishedPulling="2026-05-27 21:08:56.607340398 +0000 UTC m=+163.245965545" observedRunningTime="2026-05-27 21:08:57.412423407 +0000 UTC m=+164.051048585" watchObservedRunningTime="2026-05-27 21:08:57.412691181 +0000 UTC m=+164.051316344"
May 27 21:08:57.634437 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:57.634317    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr" podUID="377fa01a-e9e9-49a6-8640-ee79bd8c7d51" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.10:8000/healthz\": dial tcp 10.132.0.10:8000: connect: connection refused"
May 27 21:08:57.672831 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:57.672789    2570 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2"
May 27 21:08:58.386248 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:58.386216    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ds85j" event={"ID":"162181ca-487b-47a0-b0e4-911370859701","Type":"ContainerStarted","Data":"3ce5a7c5cc4e63ca0e0e2166de787382594200bb9d0dbd69a0a5a4f9114b69f3"}
May 27 21:08:58.387659 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:58.387633    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4d9hh" event={"ID":"41d38e9c-e685-40bc-a1fc-74d393746458","Type":"ContainerStarted","Data":"f06874efcc5044d84d5e1eb79dae879d9e64ca3888246306d98153b7bf701b6e"}
May 27 21:08:58.387735 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:58.387667    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4d9hh" event={"ID":"41d38e9c-e685-40bc-a1fc-74d393746458","Type":"ContainerStarted","Data":"3b32b291a77f3510914951c78ef3af43c86e04544a87506ef1c9176939fa1def"}
May 27 21:08:58.387784 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:58.387767    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4d9hh"
May 27 21:08:58.388756 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:58.388729    2570 generic.go:358] "Generic (PLEG): container finished" podID="377fa01a-e9e9-49a6-8640-ee79bd8c7d51" containerID="19764fe11251e5499f127d0232ce685eec7bae09aaf656fea241796c2d7de326" exitCode=1
May 27 21:08:58.388841 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:58.388780    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr" event={"ID":"377fa01a-e9e9-49a6-8640-ee79bd8c7d51","Type":"ContainerDied","Data":"19764fe11251e5499f127d0232ce685eec7bae09aaf656fea241796c2d7de326"}
May 27 21:08:58.389050 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:58.389036    2570 scope.go:117] "RemoveContainer" containerID="19764fe11251e5499f127d0232ce685eec7bae09aaf656fea241796c2d7de326"
May 27 21:08:58.390358 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:58.390336    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-574474d94d-wcdn2" event={"ID":"7b31b724-5935-4809-b314-ce8f4809c8aa","Type":"ContainerStarted","Data":"ec72290ddcd166653457e970329ad0991e02e694f37c9ebd3fbca81dfbdf28ea"}
May 27 21:08:58.402632 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:58.402583    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ds85j" podStartSLOduration=129.241789724 podStartE2EDuration="2m11.402550017s" podCreationTimestamp="2026-05-27 21:06:47 +0000 UTC" firstStartedPulling="2026-05-27 21:08:55.737448531 +0000 UTC m=+162.376073676" lastFinishedPulling="2026-05-27 21:08:57.898208796 +0000 UTC m=+164.536833969" observedRunningTime="2026-05-27 21:08:58.401783847 +0000 UTC m=+165.040409012" watchObservedRunningTime="2026-05-27 21:08:58.402550017 +0000 UTC m=+165.041175182"
May 27 21:08:58.455279 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:58.455229    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-4d9hh" podStartSLOduration=129.278347263 podStartE2EDuration="2m11.455214294s" podCreationTimestamp="2026-05-27 21:06:47 +0000 UTC" firstStartedPulling="2026-05-27 21:08:55.718525083 +0000 UTC m=+162.357150230" lastFinishedPulling="2026-05-27 21:08:57.895392111 +0000 UTC m=+164.534017261" observedRunningTime="2026-05-27 21:08:58.454370113 +0000 UTC m=+165.092995279" watchObservedRunningTime="2026-05-27 21:08:58.455214294 +0000 UTC m=+165.093839461"
May 27 21:08:59.394877 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:59.394836    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr" event={"ID":"377fa01a-e9e9-49a6-8640-ee79bd8c7d51","Type":"ContainerStarted","Data":"fc221eeebc4e3401a732e19355000b9f655bce517ddab6dc8e9dd84f7b487cee"}
May 27 21:08:59.395292 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:59.395130    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"
May 27 21:08:59.399871 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:08:59.399847    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84547454c5-mjnnr"
May 27 21:09:03.294099 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.294053    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l2jgs"]
May 27 21:09:03.296834 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.296809    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.300169 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.300143    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 27 21:09:03.300169 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.300154    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 27 21:09:03.300758 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.300737    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 27 21:09:03.300965 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.300949    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-krpk2\""
May 27 21:09:03.301013 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.300970    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 27 21:09:03.301013 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.300976    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 27 21:09:03.301131 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.301112    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 27 21:09:03.465461 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.465428    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-node-exporter-textfile\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.465692 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.465478    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.465692 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.465512    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-sys\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.465692 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.465541    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.465692 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.465588    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd5c8\" (UniqueName: \"kubernetes.io/projected/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-kube-api-access-nd5c8\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.465692 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.465619    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-node-exporter-wtmp\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.465692 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.465649    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-root\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.465908 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.465724    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-node-exporter-tls\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.465908 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.465760    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-metrics-client-ca\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.566511 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.566409    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-root\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.566511 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.566461    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-node-exporter-tls\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.566511 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.566493    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-metrics-client-ca\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.566806 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.566521    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-root\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.566806 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.566546    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-node-exporter-textfile\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.566806 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.566621    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.566806 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.566662    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-sys\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.566806 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.566689    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.566806 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.566716    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nd5c8\" (UniqueName: \"kubernetes.io/projected/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-kube-api-access-nd5c8\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.566806 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.566749    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-node-exporter-wtmp\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.566806 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.566765    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-sys\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.567159 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.566900    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-node-exporter-wtmp\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.567159 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.566929    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-node-exporter-textfile\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.567240 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.567206    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-metrics-client-ca\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.567296 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.567243    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.569007 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.568988    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-node-exporter-tls\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.569105 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.569063    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.582278 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.582251    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd5c8\" (UniqueName: \"kubernetes.io/projected/9a4d1aa1-8313-44ae-98c0-343b8cacd00b-kube-api-access-nd5c8\") pod \"node-exporter-l2jgs\" (UID: \"9a4d1aa1-8313-44ae-98c0-343b8cacd00b\") " pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.606111 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:03.606079    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l2jgs"
May 27 21:09:03.613759 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:09:03.613731    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4d1aa1_8313_44ae_98c0_343b8cacd00b.slice/crio-ad73150d112f30292551e9b4aa25ba3f541175686612e9a60863664d613769b3 WatchSource:0}: Error finding container ad73150d112f30292551e9b4aa25ba3f541175686612e9a60863664d613769b3: Status 404 returned error can't find the container with id ad73150d112f30292551e9b4aa25ba3f541175686612e9a60863664d613769b3
May 27 21:09:04.410712 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:04.410672    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2jgs" event={"ID":"9a4d1aa1-8313-44ae-98c0-343b8cacd00b","Type":"ContainerStarted","Data":"ad73150d112f30292551e9b4aa25ba3f541175686612e9a60863664d613769b3"}
May 27 21:09:04.930208 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:04.930115    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:09:05.414162 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:05.414130    2570 generic.go:358] "Generic (PLEG): container finished" podID="9a4d1aa1-8313-44ae-98c0-343b8cacd00b" containerID="a5a5ea98326e2b73fcb3da1f27fa1a1a864aec36d238f1ac661c1f51a848f06a" exitCode=0
May 27 21:09:05.414618 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:05.414197    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2jgs" event={"ID":"9a4d1aa1-8313-44ae-98c0-343b8cacd00b","Type":"ContainerDied","Data":"a5a5ea98326e2b73fcb3da1f27fa1a1a864aec36d238f1ac661c1f51a848f06a"}
May 27 21:09:06.418798 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:06.418764    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2jgs" event={"ID":"9a4d1aa1-8313-44ae-98c0-343b8cacd00b","Type":"ContainerStarted","Data":"418b20eedebd1f7d19dd972fffe418eae73eacef80c60e058b457f61212b461a"}
May 27 21:09:06.418798 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:06.418800    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2jgs" event={"ID":"9a4d1aa1-8313-44ae-98c0-343b8cacd00b","Type":"ContainerStarted","Data":"9fed2a558dbdf7f6e726f692fba388047c09dc924ddf0625f6edebeb94e1999b"}
May 27 21:09:06.440288 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:06.440240    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l2jgs" podStartSLOduration=2.422560706 podStartE2EDuration="3.440222042s" podCreationTimestamp="2026-05-27 21:09:03 +0000 UTC" firstStartedPulling="2026-05-27 21:09:03.615545978 +0000 UTC m=+170.254171120" lastFinishedPulling="2026-05-27 21:09:04.633207311 +0000 UTC m=+171.271832456" observedRunningTime="2026-05-27 21:09:06.438623922 +0000 UTC m=+173.077249087" watchObservedRunningTime="2026-05-27 21:09:06.440222042 +0000 UTC m=+173.078847244"
May 27 21:09:08.397991 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:08.397961    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4d9hh"
May 27 21:09:15.462711 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:15.462675    2570 patch_prober.go:28] interesting pod/image-registry-85c7f9878c-zxgmf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 27 21:09:15.463178 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:15.462732    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf" podUID="6defcd2e-1e8b-4f5f-abe8-16f4cea4af40" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 27 21:09:17.259846 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:17.259814    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85c7f9878c-zxgmf"]
May 27 21:09:17.263735 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:17.263708    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:09:42.278423 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.278374    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf" podUID="6defcd2e-1e8b-4f5f-abe8-16f4cea4af40" containerName="registry" containerID="cri-o://31983137d2d61a2308356d9762e7daca266186072b35e0add906f6b0fee36b87" gracePeriod=30
May 27 21:09:42.529169 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.529081    2570 generic.go:358] "Generic (PLEG): container finished" podID="6defcd2e-1e8b-4f5f-abe8-16f4cea4af40" containerID="31983137d2d61a2308356d9762e7daca266186072b35e0add906f6b0fee36b87" exitCode=0
May 27 21:09:42.529169 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.529149    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf" event={"ID":"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40","Type":"ContainerDied","Data":"31983137d2d61a2308356d9762e7daca266186072b35e0add906f6b0fee36b87"}
May 27 21:09:42.529333 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.529177    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf" event={"ID":"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40","Type":"ContainerDied","Data":"6192fec2e8c02db350a19422917683001894dba36ff0a85b4a218b7fabfdc858"}
May 27 21:09:42.529333 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.529190    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6192fec2e8c02db350a19422917683001894dba36ff0a85b4a218b7fabfdc858"
May 27 21:09:42.533318 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.533297    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:09:42.683268 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.683231    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-trusted-ca\") pod \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") "
May 27 21:09:42.683455 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.683281    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-certificates\") pod \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") "
May 27 21:09:42.683455 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.683310    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-ca-trust-extracted\") pod \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") "
May 27 21:09:42.683455 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.683351    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls\") pod \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") "
May 27 21:09:42.683455 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.683386    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-installation-pull-secrets\") pod \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") "
May 27 21:09:42.683455 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.683421    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-image-registry-private-configuration\") pod \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") "
May 27 21:09:42.683455 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.683454    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-bound-sa-token\") pod \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") "
May 27 21:09:42.683771 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.683495    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp2tw\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-kube-api-access-tp2tw\") pod \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\" (UID: \"6defcd2e-1e8b-4f5f-abe8-16f4cea4af40\") "
May 27 21:09:42.683825 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.683778    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 21:09:42.683872 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.683788    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 21:09:42.685887 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.685849    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 21:09:42.686045 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.686013    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 21:09:42.686218 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.686183    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 21:09:42.686299 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.686236    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 27 21:09:42.686365 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.686345    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-kube-api-access-tp2tw" (OuterVolumeSpecName: "kube-api-access-tp2tw") pod "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40"). InnerVolumeSpecName "kube-api-access-tp2tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 21:09:42.692035 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.692015    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40" (UID: "6defcd2e-1e8b-4f5f-abe8-16f4cea4af40"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 21:09:42.784603 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.784504    2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-tls\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:09:42.784603 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.784532    2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-installation-pull-secrets\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:09:42.784603 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.784542    2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-image-registry-private-configuration\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:09:42.784603 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.784554    2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-bound-sa-token\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:09:42.784603 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.784574    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp2tw\" (UniqueName: \"kubernetes.io/projected/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-kube-api-access-tp2tw\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:09:42.784603 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.784582    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-trusted-ca\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:09:42.784603 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.784592    2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-registry-certificates\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:09:42.784603 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:42.784600    2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40-ca-trust-extracted\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:09:43.531675 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:43.531646    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c7f9878c-zxgmf"
May 27 21:09:43.554520 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:43.554494    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85c7f9878c-zxgmf"]
May 27 21:09:43.563244 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:43.563221    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85c7f9878c-zxgmf"]
May 27 21:09:43.932817 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:43.932779    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6defcd2e-1e8b-4f5f-abe8-16f4cea4af40" path="/var/lib/kubelet/pods/6defcd2e-1e8b-4f5f-abe8-16f4cea4af40/volumes"
May 27 21:09:47.663367 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:47.663322    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz" podUID="3ea42c8f-88df-47ae-b348-8f52da40962a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 27 21:09:57.663046 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:09:57.663004    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz" podUID="3ea42c8f-88df-47ae-b348-8f52da40962a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 27 21:10:07.662941 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:07.662900    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz" podUID="3ea42c8f-88df-47ae-b348-8f52da40962a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 27 21:10:07.663302 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:07.662981    2570 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz"
May 27 21:10:07.663455 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:07.663428    2570 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"fdec68b137abfa12d34342a0fd764945934eb53e80cdf87cc2663277381ca2a1"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz" containerMessage="Container service-proxy failed liveness probe, will be restarted"
May 27 21:10:07.663493 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:07.663475    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz" podUID="3ea42c8f-88df-47ae-b348-8f52da40962a" containerName="service-proxy" containerID="cri-o://fdec68b137abfa12d34342a0fd764945934eb53e80cdf87cc2663277381ca2a1" gracePeriod=30
May 27 21:10:08.600574 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:08.600527    2570 generic.go:358] "Generic (PLEG): container finished" podID="3ea42c8f-88df-47ae-b348-8f52da40962a" containerID="fdec68b137abfa12d34342a0fd764945934eb53e80cdf87cc2663277381ca2a1" exitCode=2
May 27 21:10:08.600725 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:08.600583    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz" event={"ID":"3ea42c8f-88df-47ae-b348-8f52da40962a","Type":"ContainerDied","Data":"fdec68b137abfa12d34342a0fd764945934eb53e80cdf87cc2663277381ca2a1"}
May 27 21:10:08.600725 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:08.600614    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8466c6768b-gfjhz" event={"ID":"3ea42c8f-88df-47ae-b348-8f52da40962a","Type":"ContainerStarted","Data":"77221b52ce5e80dbf8c298187e53d496886731d2b470f0b69ff542c83ffa6b6c"}
May 27 21:10:25.910015 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:25.909972    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs\") pod \"network-metrics-daemon-nbt2c\" (UID: \"375b9e24-ceeb-431a-9102-ca5ac59437cb\") " pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:10:25.912441 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:25.912417    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/375b9e24-ceeb-431a-9102-ca5ac59437cb-metrics-certs\") pod \"network-metrics-daemon-nbt2c\" (UID: \"375b9e24-ceeb-431a-9102-ca5ac59437cb\") " pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:10:25.933219 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:25.933192    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nvnhn\""
May 27 21:10:25.940759 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:25.940741    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nbt2c"
May 27 21:10:26.062421 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:26.062388    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nbt2c"]
May 27 21:10:26.066386 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:10:26.066347    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375b9e24_ceeb_431a_9102_ca5ac59437cb.slice/crio-5c9176ba2caf40f20105a2d4bb3283b3fd3ece203a7bf569ecbdc53dfe43c583 WatchSource:0}: Error finding container 5c9176ba2caf40f20105a2d4bb3283b3fd3ece203a7bf569ecbdc53dfe43c583: Status 404 returned error can't find the container with id 5c9176ba2caf40f20105a2d4bb3283b3fd3ece203a7bf569ecbdc53dfe43c583
May 27 21:10:26.648468 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:26.648430    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nbt2c" event={"ID":"375b9e24-ceeb-431a-9102-ca5ac59437cb","Type":"ContainerStarted","Data":"5c9176ba2caf40f20105a2d4bb3283b3fd3ece203a7bf569ecbdc53dfe43c583"}
May 27 21:10:27.652544 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:27.652504    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nbt2c" event={"ID":"375b9e24-ceeb-431a-9102-ca5ac59437cb","Type":"ContainerStarted","Data":"578ca6463d20b904561317b4704add3b5370aeaaaa0f1270fda8e882204bfec4"}
May 27 21:10:27.652544 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:27.652544    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nbt2c" event={"ID":"375b9e24-ceeb-431a-9102-ca5ac59437cb","Type":"ContainerStarted","Data":"148e4476b8a90e0c6f6a8c81898613c4948b41124448a3ebb1805c4b4005b87b"}
May 27 21:10:27.671129 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:10:27.671079    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nbt2c" podStartSLOduration=252.717301504 podStartE2EDuration="4m13.671064469s" podCreationTimestamp="2026-05-27 21:06:14 +0000 UTC" firstStartedPulling="2026-05-27 21:10:26.068266708 +0000 UTC m=+252.706891850" lastFinishedPulling="2026-05-27 21:10:27.022029673 +0000 UTC m=+253.660654815" observedRunningTime="2026-05-27 21:10:27.670930166 +0000 UTC m=+254.309555331" watchObservedRunningTime="2026-05-27 21:10:27.671064469 +0000 UTC m=+254.309689633"
May 27 21:11:13.797009 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:11:13.796982    2570 kubelet.go:1628] "Image garbage collection succeeded"
May 27 21:12:14.552191 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.552159    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-79c8d999ff-dgws4"]
May 27 21:12:14.552594 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.552485    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6defcd2e-1e8b-4f5f-abe8-16f4cea4af40" containerName="registry"
May 27 21:12:14.552594 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.552503    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6defcd2e-1e8b-4f5f-abe8-16f4cea4af40" containerName="registry"
May 27 21:12:14.552665 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.552598    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="6defcd2e-1e8b-4f5f-abe8-16f4cea4af40" containerName="registry"
May 27 21:12:14.555240 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.555226    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-dgws4"
May 27 21:12:14.557678 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.557656    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-drjdp\""
May 27 21:12:14.557678 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.557656    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
May 27 21:12:14.558400 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.558379    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
May 27 21:12:14.565013 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.564983    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-dgws4"]
May 27 21:12:14.565120 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.565101    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvfl\" (UniqueName: \"kubernetes.io/projected/cbe75fce-5a93-445e-ab93-e49d0ec2a27e-kube-api-access-hwvfl\") pod \"cert-manager-79c8d999ff-dgws4\" (UID: \"cbe75fce-5a93-445e-ab93-e49d0ec2a27e\") " pod="cert-manager/cert-manager-79c8d999ff-dgws4"
May 27 21:12:14.565180 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.565133    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbe75fce-5a93-445e-ab93-e49d0ec2a27e-bound-sa-token\") pod \"cert-manager-79c8d999ff-dgws4\" (UID: \"cbe75fce-5a93-445e-ab93-e49d0ec2a27e\") " pod="cert-manager/cert-manager-79c8d999ff-dgws4"
May 27 21:12:14.666332 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.666296    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvfl\" (UniqueName: \"kubernetes.io/projected/cbe75fce-5a93-445e-ab93-e49d0ec2a27e-kube-api-access-hwvfl\") pod \"cert-manager-79c8d999ff-dgws4\" (UID: \"cbe75fce-5a93-445e-ab93-e49d0ec2a27e\") " pod="cert-manager/cert-manager-79c8d999ff-dgws4"
May 27 21:12:14.666487 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.666342    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbe75fce-5a93-445e-ab93-e49d0ec2a27e-bound-sa-token\") pod \"cert-manager-79c8d999ff-dgws4\" (UID: \"cbe75fce-5a93-445e-ab93-e49d0ec2a27e\") " pod="cert-manager/cert-manager-79c8d999ff-dgws4"
May 27 21:12:14.675867 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.675841    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbe75fce-5a93-445e-ab93-e49d0ec2a27e-bound-sa-token\") pod \"cert-manager-79c8d999ff-dgws4\" (UID: \"cbe75fce-5a93-445e-ab93-e49d0ec2a27e\") " pod="cert-manager/cert-manager-79c8d999ff-dgws4"
May 27 21:12:14.676647 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.676628    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvfl\" (UniqueName: \"kubernetes.io/projected/cbe75fce-5a93-445e-ab93-e49d0ec2a27e-kube-api-access-hwvfl\") pod \"cert-manager-79c8d999ff-dgws4\" (UID: \"cbe75fce-5a93-445e-ab93-e49d0ec2a27e\") " pod="cert-manager/cert-manager-79c8d999ff-dgws4"
May 27 21:12:14.864772 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.864689    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-dgws4"
May 27 21:12:14.983784 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.983747    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-dgws4"]
May 27 21:12:14.986370 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:12:14.986343    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbe75fce_5a93_445e_ab93_e49d0ec2a27e.slice/crio-f7534489183defd2fd8e55d285ec0605ad8fae87707a14a3e76dc77d0ab1622f WatchSource:0}: Error finding container f7534489183defd2fd8e55d285ec0605ad8fae87707a14a3e76dc77d0ab1622f: Status 404 returned error can't find the container with id f7534489183defd2fd8e55d285ec0605ad8fae87707a14a3e76dc77d0ab1622f
May 27 21:12:14.988051 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:14.988036    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 27 21:12:15.928625 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:15.928588    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-dgws4" event={"ID":"cbe75fce-5a93-445e-ab93-e49d0ec2a27e","Type":"ContainerStarted","Data":"f7534489183defd2fd8e55d285ec0605ad8fae87707a14a3e76dc77d0ab1622f"}
May 27 21:12:17.934248 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:17.934215    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-dgws4" event={"ID":"cbe75fce-5a93-445e-ab93-e49d0ec2a27e","Type":"ContainerStarted","Data":"af28e4f3ae50b97895aa31bb3c0f6220fa910a94d239b70783d6fc57a831ae2c"}
May 27 21:12:17.951241 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:17.951188    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-79c8d999ff-dgws4" podStartSLOduration=1.642392402 podStartE2EDuration="3.95115259s" podCreationTimestamp="2026-05-27 21:12:14 +0000 UTC" firstStartedPulling="2026-05-27 21:12:14.988152828 +0000 UTC m=+361.626777970" lastFinishedPulling="2026-05-27 21:12:17.296913016 +0000 UTC m=+363.935538158" observedRunningTime="2026-05-27 21:12:17.950025445 +0000 UTC m=+364.588650619" watchObservedRunningTime="2026-05-27 21:12:17.95115259 +0000 UTC m=+364.589777752"
May 27 21:12:40.391033 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.390999    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"]
May 27 21:12:40.393829 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.393814    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"
May 27 21:12:40.396279 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.396255    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
May 27 21:12:40.396411 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.396270    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xq28h\""
May 27 21:12:40.396487 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.396443    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
May 27 21:12:40.396586 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.396552    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
May 27 21:12:40.397095 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.397082    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
May 27 21:12:40.405608 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.405585    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"]
May 27 21:12:40.443261 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.443230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtsvn\" (UniqueName: \"kubernetes.io/projected/835def55-9d49-48a1-8c71-3755952cda77-kube-api-access-dtsvn\") pod \"opendatahub-operator-controller-manager-6678cd66dc-mk4qm\" (UID: \"835def55-9d49-48a1-8c71-3755952cda77\") " pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"
May 27 21:12:40.443398 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.443273    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/835def55-9d49-48a1-8c71-3755952cda77-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6678cd66dc-mk4qm\" (UID: \"835def55-9d49-48a1-8c71-3755952cda77\") " pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"
May 27 21:12:40.443398 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.443356    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/835def55-9d49-48a1-8c71-3755952cda77-webhook-cert\") pod \"opendatahub-operator-controller-manager-6678cd66dc-mk4qm\" (UID: \"835def55-9d49-48a1-8c71-3755952cda77\") " pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"
May 27 21:12:40.544528 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.544494    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/835def55-9d49-48a1-8c71-3755952cda77-webhook-cert\") pod \"opendatahub-operator-controller-manager-6678cd66dc-mk4qm\" (UID: \"835def55-9d49-48a1-8c71-3755952cda77\") " pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"
May 27 21:12:40.544528 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.544529    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dtsvn\" (UniqueName: \"kubernetes.io/projected/835def55-9d49-48a1-8c71-3755952cda77-kube-api-access-dtsvn\") pod \"opendatahub-operator-controller-manager-6678cd66dc-mk4qm\" (UID: \"835def55-9d49-48a1-8c71-3755952cda77\") " pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"
May 27 21:12:40.544740 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.544705    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/835def55-9d49-48a1-8c71-3755952cda77-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6678cd66dc-mk4qm\" (UID: \"835def55-9d49-48a1-8c71-3755952cda77\") " pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"
May 27 21:12:40.546914 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.546891    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/835def55-9d49-48a1-8c71-3755952cda77-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6678cd66dc-mk4qm\" (UID: \"835def55-9d49-48a1-8c71-3755952cda77\") " pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"
May 27 21:12:40.547058 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.547038    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/835def55-9d49-48a1-8c71-3755952cda77-webhook-cert\") pod \"opendatahub-operator-controller-manager-6678cd66dc-mk4qm\" (UID: \"835def55-9d49-48a1-8c71-3755952cda77\") " pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"
May 27 21:12:40.559250 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.559220    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtsvn\" (UniqueName: \"kubernetes.io/projected/835def55-9d49-48a1-8c71-3755952cda77-kube-api-access-dtsvn\") pod \"opendatahub-operator-controller-manager-6678cd66dc-mk4qm\" (UID: \"835def55-9d49-48a1-8c71-3755952cda77\") " pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"
May 27 21:12:40.704287 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.704209    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"
May 27 21:12:40.827204 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.827168    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"]
May 27 21:12:40.831369 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:12:40.831339    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835def55_9d49_48a1_8c71_3755952cda77.slice/crio-fcba1056ed117758f887294fb4faa96cd802b3cf50eba362875d9f46a40db893 WatchSource:0}: Error finding container fcba1056ed117758f887294fb4faa96cd802b3cf50eba362875d9f46a40db893: Status 404 returned error can't find the container with id fcba1056ed117758f887294fb4faa96cd802b3cf50eba362875d9f46a40db893
May 27 21:12:40.992084 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:40.991994    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm" event={"ID":"835def55-9d49-48a1-8c71-3755952cda77","Type":"ContainerStarted","Data":"fcba1056ed117758f887294fb4faa96cd802b3cf50eba362875d9f46a40db893"}
May 27 21:12:44.002625 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:44.002589    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm" event={"ID":"835def55-9d49-48a1-8c71-3755952cda77","Type":"ContainerStarted","Data":"467a4ea549ef427f504506c94a4c84b100767d9c31af7bc07d73c0496eccde14"}
May 27 21:12:44.003006 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:44.002747    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"
May 27 21:12:44.028015 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:44.027968    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm" podStartSLOduration=1.548227969 podStartE2EDuration="4.027954108s" podCreationTimestamp="2026-05-27 21:12:40 +0000 UTC" firstStartedPulling="2026-05-27 21:12:40.833006207 +0000 UTC m=+387.471631349" lastFinishedPulling="2026-05-27 21:12:43.312732342 +0000 UTC m=+389.951357488" observedRunningTime="2026-05-27 21:12:44.026326246 +0000 UTC m=+390.664951409" watchObservedRunningTime="2026-05-27 21:12:44.027954108 +0000 UTC m=+390.666579272"
May 27 21:12:55.006612 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:55.006586    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6678cd66dc-mk4qm"
May 27 21:12:58.525687 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.525650    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f"]
May 27 21:12:58.528701 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.528676    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f"
May 27 21:12:58.532769 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.532744    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 27 21:12:58.532769 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.532758    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
May 27 21:12:58.532962 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.532792    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
May 27 21:12:58.532962 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.532755    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-74vs5\""
May 27 21:12:58.532962 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.532859    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 27 21:12:58.539461 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.539441    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f"]
May 27 21:12:58.571337 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.571315    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae-tls-certs\") pod \"kube-auth-proxy-6f6f4b4c58-84r4f\" (UID: \"2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae\") " pod="openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f"
May 27 21:12:58.571460 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.571343    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27p6w\" (UniqueName: \"kubernetes.io/projected/2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae-kube-api-access-27p6w\") pod \"kube-auth-proxy-6f6f4b4c58-84r4f\" (UID: \"2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae\") " pod="openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f"
May 27 21:12:58.571460 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.571379    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae-tmp\") pod \"kube-auth-proxy-6f6f4b4c58-84r4f\" (UID: \"2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae\") " pod="openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f"
May 27 21:12:58.671760 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.671728    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae-tls-certs\") pod \"kube-auth-proxy-6f6f4b4c58-84r4f\" (UID: \"2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae\") " pod="openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f"
May 27 21:12:58.671914 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.671770    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-27p6w\" (UniqueName: \"kubernetes.io/projected/2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae-kube-api-access-27p6w\") pod \"kube-auth-proxy-6f6f4b4c58-84r4f\" (UID: \"2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae\") " pod="openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f"
May 27 21:12:58.671914 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.671803    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae-tmp\") pod \"kube-auth-proxy-6f6f4b4c58-84r4f\" (UID: \"2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae\") " pod="openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f"
May 27 21:12:58.673975 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.673948    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae-tmp\") pod \"kube-auth-proxy-6f6f4b4c58-84r4f\" (UID: \"2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae\") " pod="openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f"
May 27 21:12:58.674287 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.674266    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae-tls-certs\") pod \"kube-auth-proxy-6f6f4b4c58-84r4f\" (UID: \"2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae\") " pod="openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f"
May 27 21:12:58.680590 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.680551    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27p6w\" (UniqueName: \"kubernetes.io/projected/2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae-kube-api-access-27p6w\") pod \"kube-auth-proxy-6f6f4b4c58-84r4f\" (UID: \"2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae\") " pod="openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f"
May 27 21:12:58.838273 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.838189    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f"
May 27 21:12:58.966884 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:58.966856    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f"]
May 27 21:12:58.969232 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:12:58.969203    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2084cf02_f2c8_4b63_a1f0_9934e9a8a2ae.slice/crio-e46a01c47b10e05f2ea499993ba5007c15659f201370a2aaa5cbdfb802696b43 WatchSource:0}: Error finding container e46a01c47b10e05f2ea499993ba5007c15659f201370a2aaa5cbdfb802696b43: Status 404 returned error can't find the container with id e46a01c47b10e05f2ea499993ba5007c15659f201370a2aaa5cbdfb802696b43
May 27 21:12:59.044059 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:12:59.044022    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f" event={"ID":"2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae","Type":"ContainerStarted","Data":"e46a01c47b10e05f2ea499993ba5007c15659f201370a2aaa5cbdfb802696b43"}
May 27 21:13:01.603483 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:01.603448    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-858dbf95b8-2jzbs"]
May 27 21:13:01.606582 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:01.606546    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs"
May 27 21:13:01.609536 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:01.609507    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
May 27 21:13:01.609758 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:01.609744    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-hhf2v\""
May 27 21:13:01.621395 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:01.621373    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-858dbf95b8-2jzbs"]
May 27 21:13:01.695664 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:01.695629    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5dkt\" (UniqueName: \"kubernetes.io/projected/40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7-kube-api-access-w5dkt\") pod \"odh-model-controller-858dbf95b8-2jzbs\" (UID: \"40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7\") " pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs"
May 27 21:13:01.695823 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:01.695668    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7-cert\") pod \"odh-model-controller-858dbf95b8-2jzbs\" (UID: \"40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7\") " pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs"
May 27 21:13:01.796254 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:01.796222    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w5dkt\" (UniqueName: \"kubernetes.io/projected/40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7-kube-api-access-w5dkt\") pod \"odh-model-controller-858dbf95b8-2jzbs\" (UID: \"40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7\") " pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs"
May 27 21:13:01.796423 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:01.796261    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7-cert\") pod \"odh-model-controller-858dbf95b8-2jzbs\" (UID: \"40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7\") " pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs"
May 27 21:13:01.796423 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:13:01.796416    2570 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
May 27 21:13:01.796538 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:13:01.796492    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7-cert podName:40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7 nodeName:}" failed. No retries permitted until 2026-05-27 21:13:02.296469938 +0000 UTC m=+408.935095084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7-cert") pod "odh-model-controller-858dbf95b8-2jzbs" (UID: "40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7") : secret "odh-model-controller-webhook-cert" not found
May 27 21:13:01.808251 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:01.808222    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5dkt\" (UniqueName: \"kubernetes.io/projected/40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7-kube-api-access-w5dkt\") pod \"odh-model-controller-858dbf95b8-2jzbs\" (UID: \"40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7\") " pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs"
May 27 21:13:02.300625 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:02.300584    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7-cert\") pod \"odh-model-controller-858dbf95b8-2jzbs\" (UID: \"40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7\") " pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs"
May 27 21:13:02.300825 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:13:02.300705    2570 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
May 27 21:13:02.300825 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:13:02.300788    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7-cert podName:40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7 nodeName:}" failed. No retries permitted until 2026-05-27 21:13:03.300767686 +0000 UTC m=+409.939392829 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7-cert") pod "odh-model-controller-858dbf95b8-2jzbs" (UID: "40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7") : secret "odh-model-controller-webhook-cert" not found
May 27 21:13:03.055897 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:03.055862    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f" event={"ID":"2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae","Type":"ContainerStarted","Data":"9b61508cb5b737ad925a8342c72d1b7c7955aad3063d2525996a19d5cbdb4a1b"}
May 27 21:13:03.078152 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:03.078105    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f6f4b4c58-84r4f" podStartSLOduration=1.526642206 podStartE2EDuration="5.07808409s" podCreationTimestamp="2026-05-27 21:12:58 +0000 UTC" firstStartedPulling="2026-05-27 21:12:58.970909595 +0000 UTC m=+405.609534741" lastFinishedPulling="2026-05-27 21:13:02.522351469 +0000 UTC m=+409.160976625" observedRunningTime="2026-05-27 21:13:03.077115103 +0000 UTC m=+409.715740279" watchObservedRunningTime="2026-05-27 21:13:03.07808409 +0000 UTC m=+409.716709254"
May 27 21:13:03.308670 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:03.308571    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7-cert\") pod \"odh-model-controller-858dbf95b8-2jzbs\" (UID: \"40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7\") " pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs"
May 27 21:13:03.310908 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:03.310883    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7-cert\") pod \"odh-model-controller-858dbf95b8-2jzbs\" (UID: \"40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7\") " pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs"
May 27 21:13:03.416527 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:03.416487    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs"
May 27 21:13:03.538912 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:03.538812    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-858dbf95b8-2jzbs"]
May 27 21:13:03.541079 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:13:03.541049    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e8e370_eb87_4e57_bcf1_ff8c0f13cfb7.slice/crio-c1e240526d7a34d2ce1eb24fefcdca944e5fee49de29f752f58ed3984af88cee WatchSource:0}: Error finding container c1e240526d7a34d2ce1eb24fefcdca944e5fee49de29f752f58ed3984af88cee: Status 404 returned error can't find the container with id c1e240526d7a34d2ce1eb24fefcdca944e5fee49de29f752f58ed3984af88cee
May 27 21:13:04.059950 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:04.059908    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs" event={"ID":"40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7","Type":"ContainerStarted","Data":"c1e240526d7a34d2ce1eb24fefcdca944e5fee49de29f752f58ed3984af88cee"}
May 27 21:13:07.068741 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:07.068707    2570 generic.go:358] "Generic (PLEG): container finished" podID="40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7" containerID="61290ca192e7965a53b5d5ec192154a1b3a2f318e163f1d04f07eae607e0f3c4" exitCode=1
May 27 21:13:07.068741 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:07.068743    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs" event={"ID":"40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7","Type":"ContainerDied","Data":"61290ca192e7965a53b5d5ec192154a1b3a2f318e163f1d04f07eae607e0f3c4"}
May 27 21:13:07.069246 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:07.068994    2570 scope.go:117] "RemoveContainer" containerID="61290ca192e7965a53b5d5ec192154a1b3a2f318e163f1d04f07eae607e0f3c4"
May 27 21:13:07.646603 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:07.646571    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-856948b99f-bjkjp"]
May 27 21:13:07.649635 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:07.649608    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-856948b99f-bjkjp"
May 27 21:13:07.652240 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:07.652175    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
May 27 21:13:07.652353 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:07.652273    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-2z2cn\""
May 27 21:13:07.664465 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:07.660625    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-856948b99f-bjkjp"]
May 27 21:13:07.741824 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:07.741793    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7037f41c-22ab-4a51-888d-40b2c8998c25-cert\") pod \"kserve-controller-manager-856948b99f-bjkjp\" (UID: \"7037f41c-22ab-4a51-888d-40b2c8998c25\") " pod="opendatahub/kserve-controller-manager-856948b99f-bjkjp"
May 27 21:13:07.741963 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:07.741856    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhbzl\" (UniqueName: \"kubernetes.io/projected/7037f41c-22ab-4a51-888d-40b2c8998c25-kube-api-access-fhbzl\") pod \"kserve-controller-manager-856948b99f-bjkjp\" (UID: \"7037f41c-22ab-4a51-888d-40b2c8998c25\") " pod="opendatahub/kserve-controller-manager-856948b99f-bjkjp"
May 27 21:13:07.842443 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:07.842406    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7037f41c-22ab-4a51-888d-40b2c8998c25-cert\") pod \"kserve-controller-manager-856948b99f-bjkjp\" (UID: \"7037f41c-22ab-4a51-888d-40b2c8998c25\") " pod="opendatahub/kserve-controller-manager-856948b99f-bjkjp"
May 27 21:13:07.842622 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:07.842455    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fhbzl\" (UniqueName: \"kubernetes.io/projected/7037f41c-22ab-4a51-888d-40b2c8998c25-kube-api-access-fhbzl\") pod \"kserve-controller-manager-856948b99f-bjkjp\" (UID: \"7037f41c-22ab-4a51-888d-40b2c8998c25\") " pod="opendatahub/kserve-controller-manager-856948b99f-bjkjp"
May 27 21:13:07.842622 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:13:07.842547    2570 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
May 27 21:13:07.842864 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:13:07.842629    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7037f41c-22ab-4a51-888d-40b2c8998c25-cert podName:7037f41c-22ab-4a51-888d-40b2c8998c25 nodeName:}" failed. No retries permitted until 2026-05-27 21:13:08.342611789 +0000 UTC m=+414.981236941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7037f41c-22ab-4a51-888d-40b2c8998c25-cert") pod "kserve-controller-manager-856948b99f-bjkjp" (UID: "7037f41c-22ab-4a51-888d-40b2c8998c25") : secret "kserve-webhook-server-cert" not found
May 27 21:13:07.856989 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:07.856951    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhbzl\" (UniqueName: \"kubernetes.io/projected/7037f41c-22ab-4a51-888d-40b2c8998c25-kube-api-access-fhbzl\") pod \"kserve-controller-manager-856948b99f-bjkjp\" (UID: \"7037f41c-22ab-4a51-888d-40b2c8998c25\") " pod="opendatahub/kserve-controller-manager-856948b99f-bjkjp"
May 27 21:13:08.073962 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:08.073875    2570 generic.go:358] "Generic (PLEG): container finished" podID="40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7" containerID="0111fa405ccc6c0fd6e6a7a63c4e3cf8ba85e2e1107ce913ac5d3dbf8fc18607" exitCode=1
May 27 21:13:08.073962 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:08.073950    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs" event={"ID":"40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7","Type":"ContainerDied","Data":"0111fa405ccc6c0fd6e6a7a63c4e3cf8ba85e2e1107ce913ac5d3dbf8fc18607"}
May 27 21:13:08.074370 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:08.073983    2570 scope.go:117] "RemoveContainer" containerID="61290ca192e7965a53b5d5ec192154a1b3a2f318e163f1d04f07eae607e0f3c4"
May 27 21:13:08.074370 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:08.074180    2570 scope.go:117] "RemoveContainer" containerID="0111fa405ccc6c0fd6e6a7a63c4e3cf8ba85e2e1107ce913ac5d3dbf8fc18607"
May 27 21:13:08.074439 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:13:08.074367    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=odh-model-controller-858dbf95b8-2jzbs_opendatahub(40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7)\"" pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs" podUID="40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7"
May 27 21:13:08.347438 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:08.347345    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7037f41c-22ab-4a51-888d-40b2c8998c25-cert\") pod \"kserve-controller-manager-856948b99f-bjkjp\" (UID: \"7037f41c-22ab-4a51-888d-40b2c8998c25\") " pod="opendatahub/kserve-controller-manager-856948b99f-bjkjp"
May 27 21:13:08.349660 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:08.349630    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7037f41c-22ab-4a51-888d-40b2c8998c25-cert\") pod \"kserve-controller-manager-856948b99f-bjkjp\" (UID: \"7037f41c-22ab-4a51-888d-40b2c8998c25\") " pod="opendatahub/kserve-controller-manager-856948b99f-bjkjp"
May 27 21:13:08.568477 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:08.568441    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-856948b99f-bjkjp"
May 27 21:13:08.694016 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:08.693982    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-856948b99f-bjkjp"]
May 27 21:13:08.696489 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:13:08.696466    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7037f41c_22ab_4a51_888d_40b2c8998c25.slice/crio-de7f3c103a6f4117a3f72c14b5be7784351acfb0f6dbe04871fdf72a627e2016 WatchSource:0}: Error finding container de7f3c103a6f4117a3f72c14b5be7784351acfb0f6dbe04871fdf72a627e2016: Status 404 returned error can't find the container with id de7f3c103a6f4117a3f72c14b5be7784351acfb0f6dbe04871fdf72a627e2016
May 27 21:13:09.077240 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:09.077205    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-856948b99f-bjkjp" event={"ID":"7037f41c-22ab-4a51-888d-40b2c8998c25","Type":"ContainerStarted","Data":"de7f3c103a6f4117a3f72c14b5be7784351acfb0f6dbe04871fdf72a627e2016"}
May 27 21:13:09.078646 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:09.078622    2570 scope.go:117] "RemoveContainer" containerID="0111fa405ccc6c0fd6e6a7a63c4e3cf8ba85e2e1107ce913ac5d3dbf8fc18607"
May 27 21:13:09.078790 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:13:09.078776    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=odh-model-controller-858dbf95b8-2jzbs_opendatahub(40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7)\"" pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs" podUID="40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7"
May 27 21:13:13.095122 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:13.095082    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-856948b99f-bjkjp" event={"ID":"7037f41c-22ab-4a51-888d-40b2c8998c25","Type":"ContainerStarted","Data":"b54e68994cea0ddf5051e53b33aa22f892cea2696a32f3a86bad2a986024cd9f"}
May 27 21:13:13.095543 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:13.095208    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-856948b99f-bjkjp"
May 27 21:13:13.113802 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:13.113724    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-856948b99f-bjkjp" podStartSLOduration=2.5803686580000003 podStartE2EDuration="6.113708309s" podCreationTimestamp="2026-05-27 21:13:07 +0000 UTC" firstStartedPulling="2026-05-27 21:13:08.697771111 +0000 UTC m=+415.336396254" lastFinishedPulling="2026-05-27 21:13:12.231110761 +0000 UTC m=+418.869735905" observedRunningTime="2026-05-27 21:13:13.113040869 +0000 UTC m=+419.751666034" watchObservedRunningTime="2026-05-27 21:13:13.113708309 +0000 UTC m=+419.752333474"
May 27 21:13:13.416975 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:13.416941    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs"
May 27 21:13:13.417343 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:13.417330    2570 scope.go:117] "RemoveContainer" containerID="0111fa405ccc6c0fd6e6a7a63c4e3cf8ba85e2e1107ce913ac5d3dbf8fc18607"
May 27 21:13:13.417512 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:13:13.417497    2570 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=odh-model-controller-858dbf95b8-2jzbs_opendatahub(40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7)\"" pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs" podUID="40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7"
May 27 21:13:14.442960 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:14.442929    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm"]
May 27 21:13:14.446061 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:14.446045    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm"
May 27 21:13:14.450094 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:14.450070    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"kube-root-ca.crt\""
May 27 21:13:14.450206 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:14.450121    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-operators\"/\"servicemesh-operator3-dockercfg-srwtq\""
May 27 21:13:14.450444 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:14.450430    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"openshift-service-ca.crt\""
May 27 21:13:14.459301 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:14.459277    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm"]
May 27 21:13:14.597638 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:14.597602    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tks82\" (UniqueName: \"kubernetes.io/projected/af2ca4e8-90a6-4134-9d11-ef2972aad6d3-kube-api-access-tks82\") pod \"servicemesh-operator3-55f49c5f94-gtrtm\" (UID: \"af2ca4e8-90a6-4134-9d11-ef2972aad6d3\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm"
May 27 21:13:14.597819 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:14.597646    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/af2ca4e8-90a6-4134-9d11-ef2972aad6d3-operator-config\") pod \"servicemesh-operator3-55f49c5f94-gtrtm\" (UID: \"af2ca4e8-90a6-4134-9d11-ef2972aad6d3\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm"
May 27 21:13:14.698874 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:14.698796    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tks82\" (UniqueName: \"kubernetes.io/projected/af2ca4e8-90a6-4134-9d11-ef2972aad6d3-kube-api-access-tks82\") pod \"servicemesh-operator3-55f49c5f94-gtrtm\" (UID: \"af2ca4e8-90a6-4134-9d11-ef2972aad6d3\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm"
May 27 21:13:14.698874 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:14.698836    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/af2ca4e8-90a6-4134-9d11-ef2972aad6d3-operator-config\") pod \"servicemesh-operator3-55f49c5f94-gtrtm\" (UID: \"af2ca4e8-90a6-4134-9d11-ef2972aad6d3\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm"
May 27 21:13:14.701313 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:14.701289    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/af2ca4e8-90a6-4134-9d11-ef2972aad6d3-operator-config\") pod \"servicemesh-operator3-55f49c5f94-gtrtm\" (UID: \"af2ca4e8-90a6-4134-9d11-ef2972aad6d3\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm"
May 27 21:13:14.708532 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:14.708505    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tks82\" (UniqueName: \"kubernetes.io/projected/af2ca4e8-90a6-4134-9d11-ef2972aad6d3-kube-api-access-tks82\") pod \"servicemesh-operator3-55f49c5f94-gtrtm\" (UID: \"af2ca4e8-90a6-4134-9d11-ef2972aad6d3\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm"
May 27 21:13:14.754497 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:14.754461    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm"
May 27 21:13:14.913799 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:14.913746    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm"]
May 27 21:13:14.917548 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:13:14.917509    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2ca4e8_90a6_4134_9d11_ef2972aad6d3.slice/crio-e31b49b787a916e944eeca7b48a3a5f5ea482ee1dbd5d25989ae557197fdbc4f WatchSource:0}: Error finding container e31b49b787a916e944eeca7b48a3a5f5ea482ee1dbd5d25989ae557197fdbc4f: Status 404 returned error can't find the container with id e31b49b787a916e944eeca7b48a3a5f5ea482ee1dbd5d25989ae557197fdbc4f
May 27 21:13:15.102730 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:15.102647    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm" event={"ID":"af2ca4e8-90a6-4134-9d11-ef2972aad6d3","Type":"ContainerStarted","Data":"e31b49b787a916e944eeca7b48a3a5f5ea482ee1dbd5d25989ae557197fdbc4f"}
May 27 21:13:18.113953 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:18.113870    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm" event={"ID":"af2ca4e8-90a6-4134-9d11-ef2972aad6d3","Type":"ContainerStarted","Data":"7793cbdaa97395fb21bd5e8b770e54d80846856f81810329ea94796bf9bee8ff"}
May 27 21:13:18.114311 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:18.114027    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm"
May 27 21:13:18.139774 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:18.139729    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm" podStartSLOduration=1.290763431 podStartE2EDuration="4.139717638s" podCreationTimestamp="2026-05-27 21:13:14 +0000 UTC" firstStartedPulling="2026-05-27 21:13:14.919933306 +0000 UTC m=+421.558558448" lastFinishedPulling="2026-05-27 21:13:17.768887497 +0000 UTC m=+424.407512655" observedRunningTime="2026-05-27 21:13:18.138261674 +0000 UTC m=+424.776886838" watchObservedRunningTime="2026-05-27 21:13:18.139717638 +0000 UTC m=+424.778342801"
May 27 21:13:23.416975 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:23.416939    2570 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs"
May 27 21:13:23.417407 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:23.417310    2570 scope.go:117] "RemoveContainer" containerID="0111fa405ccc6c0fd6e6a7a63c4e3cf8ba85e2e1107ce913ac5d3dbf8fc18607"
May 27 21:13:24.132772 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.132742    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs" event={"ID":"40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7","Type":"ContainerStarted","Data":"53e4435b1f59bd488e5fc2e935cbd40ec17b1a095d957b12e7095eb02ed8da80"}
May 27 21:13:24.132958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.132942    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs"
May 27 21:13:24.763352 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.763296    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs" podStartSLOduration=3.453822707 podStartE2EDuration="23.763278823s" podCreationTimestamp="2026-05-27 21:13:01 +0000 UTC" firstStartedPulling="2026-05-27 21:13:03.542374216 +0000 UTC m=+410.180999372" lastFinishedPulling="2026-05-27 21:13:23.851830345 +0000 UTC m=+430.490455488" observedRunningTime="2026-05-27 21:13:24.160946836 +0000 UTC m=+430.799572000" watchObservedRunningTime="2026-05-27 21:13:24.763278823 +0000 UTC m=+431.401903989"
May 27 21:13:24.764489 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.764463    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"]
May 27 21:13:24.767518 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.767504    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.770084 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.770065    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
May 27 21:13:24.770188 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.770071    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
May 27 21:13:24.770188 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.770141    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
May 27 21:13:24.770388 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.770374    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
May 27 21:13:24.770683 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.770669    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-j7t99\""
May 27 21:13:24.779238 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.779218    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"]
May 27 21:13:24.878213 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.878178    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73350e66-fa44-49e6-aa5f-e02023fa01be-istio-token\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.878374 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.878222    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73350e66-fa44-49e6-aa5f-e02023fa01be-istio-kubeconfig\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.878374 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.878296    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/73350e66-fa44-49e6-aa5f-e02023fa01be-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.878374 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.878329    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/73350e66-fa44-49e6-aa5f-e02023fa01be-local-certs\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.878374 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.878346    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxgvc\" (UniqueName: \"kubernetes.io/projected/73350e66-fa44-49e6-aa5f-e02023fa01be-kube-api-access-fxgvc\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.878523 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.878406    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/73350e66-fa44-49e6-aa5f-e02023fa01be-cacerts\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.878523 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.878420    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/73350e66-fa44-49e6-aa5f-e02023fa01be-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.979754 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.979715    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/73350e66-fa44-49e6-aa5f-e02023fa01be-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.979950 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.979770    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/73350e66-fa44-49e6-aa5f-e02023fa01be-local-certs\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.979950 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.979797    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fxgvc\" (UniqueName: \"kubernetes.io/projected/73350e66-fa44-49e6-aa5f-e02023fa01be-kube-api-access-fxgvc\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.979950 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.979859    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/73350e66-fa44-49e6-aa5f-e02023fa01be-cacerts\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.979950 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.979884    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/73350e66-fa44-49e6-aa5f-e02023fa01be-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.979950 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.979943    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73350e66-fa44-49e6-aa5f-e02023fa01be-istio-token\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.980199 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.979974    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73350e66-fa44-49e6-aa5f-e02023fa01be-istio-kubeconfig\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.980517 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.980496    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/73350e66-fa44-49e6-aa5f-e02023fa01be-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.982464 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.982437    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/73350e66-fa44-49e6-aa5f-e02023fa01be-local-certs\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.982752 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.982729    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/73350e66-fa44-49e6-aa5f-e02023fa01be-istio-kubeconfig\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.982922 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.982900    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/73350e66-fa44-49e6-aa5f-e02023fa01be-cacerts\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.983005 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.982906    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/73350e66-fa44-49e6-aa5f-e02023fa01be-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.989095 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.989074    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxgvc\" (UniqueName: \"kubernetes.io/projected/73350e66-fa44-49e6-aa5f-e02023fa01be-kube-api-access-fxgvc\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:24.989991 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:24.989972    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73350e66-fa44-49e6-aa5f-e02023fa01be-istio-token\") pod \"istiod-openshift-gateway-798958bb55-jnxvm\" (UID: \"73350e66-fa44-49e6-aa5f-e02023fa01be\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:25.076068 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:25.075990    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:25.216821 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:25.216793    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"]
May 27 21:13:25.219742 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:13:25.219692    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73350e66_fa44_49e6_aa5f_e02023fa01be.slice/crio-a8d1ffe63d1b329f8edd91a90abfb0a1a0a02fe7282a70fabc4d40e3fba90abc WatchSource:0}: Error finding container a8d1ffe63d1b329f8edd91a90abfb0a1a0a02fe7282a70fabc4d40e3fba90abc: Status 404 returned error can't find the container with id a8d1ffe63d1b329f8edd91a90abfb0a1a0a02fe7282a70fabc4d40e3fba90abc
May 27 21:13:26.142032 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:26.141992    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm" event={"ID":"73350e66-fa44-49e6-aa5f-e02023fa01be","Type":"ContainerStarted","Data":"a8d1ffe63d1b329f8edd91a90abfb0a1a0a02fe7282a70fabc4d40e3fba90abc"}
May 27 21:13:28.148996 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:28.148960    2570 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 27 21:13:28.149207 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:28.149026    2570 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 27 21:13:29.119476 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:29.119441    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-gtrtm"
May 27 21:13:29.152603 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:29.152553    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm" event={"ID":"73350e66-fa44-49e6-aa5f-e02023fa01be","Type":"ContainerStarted","Data":"a881b69129ce40500713eed923133473babfa0e79c89fab30103d813a605a49c"}
May 27 21:13:29.153318 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:29.153295    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:29.155598 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:29.155545    2570 patch_prober.go:28] interesting pod/istiod-openshift-gateway-798958bb55-jnxvm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
May 27 21:13:29.155829 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:29.155798    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm" podUID="73350e66-fa44-49e6-aa5f-e02023fa01be" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 27 21:13:29.189470 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:29.189422    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm" podStartSLOduration=2.262485859 podStartE2EDuration="5.189407447s" podCreationTimestamp="2026-05-27 21:13:24 +0000 UTC" firstStartedPulling="2026-05-27 21:13:25.221833887 +0000 UTC m=+431.860459043" lastFinishedPulling="2026-05-27 21:13:28.148755476 +0000 UTC m=+434.787380631" observedRunningTime="2026-05-27 21:13:29.187149761 +0000 UTC m=+435.825774924" watchObservedRunningTime="2026-05-27 21:13:29.189407447 +0000 UTC m=+435.828032612"
May 27 21:13:30.156469 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:30.156441    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-jnxvm"
May 27 21:13:35.140096 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:35.140070    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-858dbf95b8-2jzbs"
May 27 21:13:44.103226 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:13:44.103188    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-856948b99f-bjkjp"
May 27 21:14:34.714791 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:34.714756    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"]
May 27 21:14:34.718045 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:34.718029    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"
May 27 21:14:34.721460 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:34.721435    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 27 21:14:34.721620 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:34.721477    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l7wxj\""
May 27 21:14:34.721620 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:34.721498    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 27 21:14:34.774485 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:34.774426    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"]
May 27 21:14:34.801731 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:34.801694    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh9m7\" (UniqueName: \"kubernetes.io/projected/2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d-kube-api-access-vh9m7\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-lsck9\" (UID: \"2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"
May 27 21:14:34.801890 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:34.801808    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-lsck9\" (UID: \"2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"
May 27 21:14:34.902340 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:34.902303    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-lsck9\" (UID: \"2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"
May 27 21:14:34.902528 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:34.902358    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vh9m7\" (UniqueName: \"kubernetes.io/projected/2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d-kube-api-access-vh9m7\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-lsck9\" (UID: \"2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"
May 27 21:14:34.902708 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:34.902685    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-lsck9\" (UID: \"2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"
May 27 21:14:34.922744 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:34.922721    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh9m7\" (UniqueName: \"kubernetes.io/projected/2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d-kube-api-access-vh9m7\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-lsck9\" (UID: \"2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"
May 27 21:14:35.027996 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:35.027917    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"
May 27 21:14:35.170886 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:35.170862    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"]
May 27 21:14:35.171477 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:14:35.171455    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4db75e_f0b8_4bf9_86b9_79ad8c97ea6d.slice/crio-bb319ba2baf3d7cc10ec9414a9bb15e8593455d3d0559e32d33948c7970a03f6 WatchSource:0}: Error finding container bb319ba2baf3d7cc10ec9414a9bb15e8593455d3d0559e32d33948c7970a03f6: Status 404 returned error can't find the container with id bb319ba2baf3d7cc10ec9414a9bb15e8593455d3d0559e32d33948c7970a03f6
May 27 21:14:35.353922 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:35.353842    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9" event={"ID":"2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d","Type":"ContainerStarted","Data":"bb319ba2baf3d7cc10ec9414a9bb15e8593455d3d0559e32d33948c7970a03f6"}
May 27 21:14:41.373169 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:41.373133    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9" event={"ID":"2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d","Type":"ContainerStarted","Data":"48a3978088e71353d59b9b0369c65d4eb89104556ed9aae6779c8095ef9e94c2"}
May 27 21:14:41.373553 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:41.373243    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"
May 27 21:14:41.398812 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:41.398737    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9" podStartSLOduration=1.8002647330000001 podStartE2EDuration="7.398719174s" podCreationTimestamp="2026-05-27 21:14:34 +0000 UTC" firstStartedPulling="2026-05-27 21:14:35.173598544 +0000 UTC m=+501.812223687" lastFinishedPulling="2026-05-27 21:14:40.772052983 +0000 UTC m=+507.410678128" observedRunningTime="2026-05-27 21:14:41.396589774 +0000 UTC m=+508.035214939" watchObservedRunningTime="2026-05-27 21:14:41.398719174 +0000 UTC m=+508.037344338"
May 27 21:14:52.377976 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:52.377944    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"
May 27 21:14:54.240351 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.240319    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"]
May 27 21:14:54.240763 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.240515    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9" podUID="2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d" containerName="manager" containerID="cri-o://48a3978088e71353d59b9b0369c65d4eb89104556ed9aae6779c8095ef9e94c2" gracePeriod=2
May 27 21:14:54.262579 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.262542    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"]
May 27 21:14:54.307810 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.307779    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"]
May 27 21:14:54.308061 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.308049    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d" containerName="manager"
May 27 21:14:54.308105 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.308063    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d" containerName="manager"
May 27 21:14:54.308138 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.308117    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d" containerName="manager"
May 27 21:14:54.310989 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.310973    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"
May 27 21:14:54.320306 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.320275    2570 status_manager.go:895] "Failed to get status for pod" podUID="2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9" err="pods \"kuadrant-operator-controller-manager-5f895dd7d5-lsck9\" is forbidden: User \"system:node:ip-10-0-128-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-93.ec2.internal' and this object"
May 27 21:14:54.347780 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.347754    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"]
May 27 21:14:54.359048 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.359019    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr4qg\" (UniqueName: \"kubernetes.io/projected/6611d9e0-cb9e-4b2f-a4b2-f29364327159-kube-api-access-nr4qg\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-vtb52\" (UID: \"6611d9e0-cb9e-4b2f-a4b2-f29364327159\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"
May 27 21:14:54.359175 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.359114    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6611d9e0-cb9e-4b2f-a4b2-f29364327159-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-vtb52\" (UID: \"6611d9e0-cb9e-4b2f-a4b2-f29364327159\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"
May 27 21:14:54.416815 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.416771    2570 generic.go:358] "Generic (PLEG): container finished" podID="2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d" containerID="48a3978088e71353d59b9b0369c65d4eb89104556ed9aae6779c8095ef9e94c2" exitCode=0
May 27 21:14:54.460433 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.460406    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6611d9e0-cb9e-4b2f-a4b2-f29364327159-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-vtb52\" (UID: \"6611d9e0-cb9e-4b2f-a4b2-f29364327159\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"
May 27 21:14:54.460527 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.460466    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nr4qg\" (UniqueName: \"kubernetes.io/projected/6611d9e0-cb9e-4b2f-a4b2-f29364327159-kube-api-access-nr4qg\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-vtb52\" (UID: \"6611d9e0-cb9e-4b2f-a4b2-f29364327159\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"
May 27 21:14:54.460820 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.460801    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6611d9e0-cb9e-4b2f-a4b2-f29364327159-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-vtb52\" (UID: \"6611d9e0-cb9e-4b2f-a4b2-f29364327159\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"
May 27 21:14:54.472815 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.472798    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"
May 27 21:14:54.474882 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.474861    2570 status_manager.go:895] "Failed to get status for pod" podUID="2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9" err="pods \"kuadrant-operator-controller-manager-5f895dd7d5-lsck9\" is forbidden: User \"system:node:ip-10-0-128-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-93.ec2.internal' and this object"
May 27 21:14:54.479213 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.479194    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr4qg\" (UniqueName: \"kubernetes.io/projected/6611d9e0-cb9e-4b2f-a4b2-f29364327159-kube-api-access-nr4qg\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-vtb52\" (UID: \"6611d9e0-cb9e-4b2f-a4b2-f29364327159\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"
May 27 21:14:54.561124 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.561053    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh9m7\" (UniqueName: \"kubernetes.io/projected/2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d-kube-api-access-vh9m7\") pod \"2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d\" (UID: \"2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d\") "
May 27 21:14:54.561124 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.561091    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d-extensions-socket-volume\") pod \"2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d\" (UID: \"2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d\") "
May 27 21:14:54.561614 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.561586    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d" (UID: "2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 21:14:54.563049 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.563028    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d-kube-api-access-vh9m7" (OuterVolumeSpecName: "kube-api-access-vh9m7") pod "2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d" (UID: "2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d"). InnerVolumeSpecName "kube-api-access-vh9m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 21:14:54.622200 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.622172    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"
May 27 21:14:54.662248 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.662216    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh9m7\" (UniqueName: \"kubernetes.io/projected/2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d-kube-api-access-vh9m7\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:14:54.662248 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.662244    2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d-extensions-socket-volume\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:14:54.759797 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:54.759769    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"]
May 27 21:14:54.763032 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:14:54.763004    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6611d9e0_cb9e_4b2f_a4b2_f29364327159.slice/crio-3f97674434e6a9c357630d356c74bd3e7c269a7f70ea67d509da3f58c1357466 WatchSource:0}: Error finding container 3f97674434e6a9c357630d356c74bd3e7c269a7f70ea67d509da3f58c1357466: Status 404 returned error can't find the container with id 3f97674434e6a9c357630d356c74bd3e7c269a7f70ea67d509da3f58c1357466
May 27 21:14:55.421425 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:55.421394    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9"
May 27 21:14:55.421838 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:55.421448    2570 scope.go:117] "RemoveContainer" containerID="48a3978088e71353d59b9b0369c65d4eb89104556ed9aae6779c8095ef9e94c2"
May 27 21:14:55.423218 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:55.423192    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52" event={"ID":"6611d9e0-cb9e-4b2f-a4b2-f29364327159","Type":"ContainerStarted","Data":"804a658e53d2d06c8bec787de0bf59cbf0cb684e8f40ee0ff1e93b7e340be9f0"}
May 27 21:14:55.423301 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:55.423230    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52" event={"ID":"6611d9e0-cb9e-4b2f-a4b2-f29364327159","Type":"ContainerStarted","Data":"3f97674434e6a9c357630d356c74bd3e7c269a7f70ea67d509da3f58c1357466"}
May 27 21:14:55.423301 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:55.423293    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"
May 27 21:14:55.434264 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:55.434243    2570 status_manager.go:895] "Failed to get status for pod" podUID="2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9" err="pods \"kuadrant-operator-controller-manager-5f895dd7d5-lsck9\" is forbidden: User \"system:node:ip-10-0-128-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-93.ec2.internal' and this object"
May 27 21:14:55.474611 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:55.474547    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52" podStartSLOduration=1.4745330669999999 podStartE2EDuration="1.474533067s" podCreationTimestamp="2026-05-27 21:14:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 21:14:55.472362078 +0000 UTC m=+522.110987265" watchObservedRunningTime="2026-05-27 21:14:55.474533067 +0000 UTC m=+522.113158231"
May 27 21:14:55.475080 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:55.475053    2570 status_manager.go:895] "Failed to get status for pod" podUID="2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-lsck9" err="pods \"kuadrant-operator-controller-manager-5f895dd7d5-lsck9\" is forbidden: User \"system:node:ip-10-0-128-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-93.ec2.internal' and this object"
May 27 21:14:55.934283 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:55.934251    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d" path="/var/lib/kubelet/pods/2a4db75e-f0b8-4bf9-86b9-79ad8c97ea6d/volumes"
May 27 21:14:56.020245 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:56.020209    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"]
May 27 21:14:56.024938 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:56.024916    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"
May 27 21:14:56.038100 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:56.038079    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"]
May 27 21:14:56.073175 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:56.073146    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02436455-741f-4d29-ae2e-dcedad32bd18-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55c7f4c975-s4p57\" (UID: \"02436455-741f-4d29-ae2e-dcedad32bd18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"
May 27 21:14:56.073297 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:56.073189    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z54x\" (UniqueName: \"kubernetes.io/projected/02436455-741f-4d29-ae2e-dcedad32bd18-kube-api-access-8z54x\") pod \"kuadrant-operator-controller-manager-55c7f4c975-s4p57\" (UID: \"02436455-741f-4d29-ae2e-dcedad32bd18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"
May 27 21:14:56.173758 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:56.173727    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02436455-741f-4d29-ae2e-dcedad32bd18-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55c7f4c975-s4p57\" (UID: \"02436455-741f-4d29-ae2e-dcedad32bd18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"
May 27 21:14:56.173899 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:56.173776    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8z54x\" (UniqueName: \"kubernetes.io/projected/02436455-741f-4d29-ae2e-dcedad32bd18-kube-api-access-8z54x\") pod \"kuadrant-operator-controller-manager-55c7f4c975-s4p57\" (UID: \"02436455-741f-4d29-ae2e-dcedad32bd18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"
May 27 21:14:56.174179 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:56.174156    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02436455-741f-4d29-ae2e-dcedad32bd18-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55c7f4c975-s4p57\" (UID: \"02436455-741f-4d29-ae2e-dcedad32bd18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"
May 27 21:14:56.194302 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:56.194242    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z54x\" (UniqueName: \"kubernetes.io/projected/02436455-741f-4d29-ae2e-dcedad32bd18-kube-api-access-8z54x\") pod \"kuadrant-operator-controller-manager-55c7f4c975-s4p57\" (UID: \"02436455-741f-4d29-ae2e-dcedad32bd18\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"
May 27 21:14:56.334210 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:56.334176    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"
May 27 21:14:56.487902 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:56.487873    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"]
May 27 21:14:56.489169 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:14:56.489131    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02436455_741f_4d29_ae2e_dcedad32bd18.slice/crio-f23f550f967a7423957e884c54d8343609bd7ddf9f63eaa980d53100a75d1977 WatchSource:0}: Error finding container f23f550f967a7423957e884c54d8343609bd7ddf9f63eaa980d53100a75d1977: Status 404 returned error can't find the container with id f23f550f967a7423957e884c54d8343609bd7ddf9f63eaa980d53100a75d1977
May 27 21:14:57.432205 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:57.432171    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57" event={"ID":"02436455-741f-4d29-ae2e-dcedad32bd18","Type":"ContainerStarted","Data":"41f300203a3a3e6b39065e985606b60a065a86df460dce7ef83d5c1a177e008d"}
May 27 21:14:57.432205 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:57.432209    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57" event={"ID":"02436455-741f-4d29-ae2e-dcedad32bd18","Type":"ContainerStarted","Data":"f23f550f967a7423957e884c54d8343609bd7ddf9f63eaa980d53100a75d1977"}
May 27 21:14:57.432403 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:57.432302    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"
May 27 21:14:57.476849 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:14:57.476804    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57" podStartSLOduration=2.476789682 podStartE2EDuration="2.476789682s" podCreationTimestamp="2026-05-27 21:14:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 21:14:57.474180675 +0000 UTC m=+524.112805839" watchObservedRunningTime="2026-05-27 21:14:57.476789682 +0000 UTC m=+524.115414846"
May 27 21:15:06.430578 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:06.430539    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"
May 27 21:15:08.438317 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:08.438291    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"
May 27 21:15:08.535123 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:08.535093    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"]
May 27 21:15:08.535333 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:08.535312    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52" podUID="6611d9e0-cb9e-4b2f-a4b2-f29364327159" containerName="manager" containerID="cri-o://804a658e53d2d06c8bec787de0bf59cbf0cb684e8f40ee0ff1e93b7e340be9f0" gracePeriod=10
May 27 21:15:08.776611 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:08.776583    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"
May 27 21:15:08.865470 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:08.865438    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6611d9e0-cb9e-4b2f-a4b2-f29364327159-extensions-socket-volume\") pod \"6611d9e0-cb9e-4b2f-a4b2-f29364327159\" (UID: \"6611d9e0-cb9e-4b2f-a4b2-f29364327159\") "
May 27 21:15:08.865649 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:08.865512    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr4qg\" (UniqueName: \"kubernetes.io/projected/6611d9e0-cb9e-4b2f-a4b2-f29364327159-kube-api-access-nr4qg\") pod \"6611d9e0-cb9e-4b2f-a4b2-f29364327159\" (UID: \"6611d9e0-cb9e-4b2f-a4b2-f29364327159\") "
May 27 21:15:08.865835 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:08.865809    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6611d9e0-cb9e-4b2f-a4b2-f29364327159-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6611d9e0-cb9e-4b2f-a4b2-f29364327159" (UID: "6611d9e0-cb9e-4b2f-a4b2-f29364327159"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 21:15:08.867369 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:08.867343    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6611d9e0-cb9e-4b2f-a4b2-f29364327159-kube-api-access-nr4qg" (OuterVolumeSpecName: "kube-api-access-nr4qg") pod "6611d9e0-cb9e-4b2f-a4b2-f29364327159" (UID: "6611d9e0-cb9e-4b2f-a4b2-f29364327159"). InnerVolumeSpecName "kube-api-access-nr4qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 21:15:08.966814 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:08.966745    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr4qg\" (UniqueName: \"kubernetes.io/projected/6611d9e0-cb9e-4b2f-a4b2-f29364327159-kube-api-access-nr4qg\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:15:08.966814 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:08.966774    2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6611d9e0-cb9e-4b2f-a4b2-f29364327159-extensions-socket-volume\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:15:09.469153 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:09.469111    2570 generic.go:358] "Generic (PLEG): container finished" podID="6611d9e0-cb9e-4b2f-a4b2-f29364327159" containerID="804a658e53d2d06c8bec787de0bf59cbf0cb684e8f40ee0ff1e93b7e340be9f0" exitCode=0
May 27 21:15:09.469537 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:09.469169    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"
May 27 21:15:09.469537 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:09.469173    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52" event={"ID":"6611d9e0-cb9e-4b2f-a4b2-f29364327159","Type":"ContainerDied","Data":"804a658e53d2d06c8bec787de0bf59cbf0cb684e8f40ee0ff1e93b7e340be9f0"}
May 27 21:15:09.469537 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:09.469200    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52" event={"ID":"6611d9e0-cb9e-4b2f-a4b2-f29364327159","Type":"ContainerDied","Data":"3f97674434e6a9c357630d356c74bd3e7c269a7f70ea67d509da3f58c1357466"}
May 27 21:15:09.469537 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:09.469214    2570 scope.go:117] "RemoveContainer" containerID="804a658e53d2d06c8bec787de0bf59cbf0cb684e8f40ee0ff1e93b7e340be9f0"
May 27 21:15:09.476824 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:09.476791    2570 scope.go:117] "RemoveContainer" containerID="804a658e53d2d06c8bec787de0bf59cbf0cb684e8f40ee0ff1e93b7e340be9f0"
May 27 21:15:09.477142 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:15:09.477120    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"804a658e53d2d06c8bec787de0bf59cbf0cb684e8f40ee0ff1e93b7e340be9f0\": container with ID starting with 804a658e53d2d06c8bec787de0bf59cbf0cb684e8f40ee0ff1e93b7e340be9f0 not found: ID does not exist" containerID="804a658e53d2d06c8bec787de0bf59cbf0cb684e8f40ee0ff1e93b7e340be9f0"
May 27 21:15:09.477216 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:09.477150    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804a658e53d2d06c8bec787de0bf59cbf0cb684e8f40ee0ff1e93b7e340be9f0"} err="failed to get container status \"804a658e53d2d06c8bec787de0bf59cbf0cb684e8f40ee0ff1e93b7e340be9f0\": rpc error: code = NotFound desc = could not find container \"804a658e53d2d06c8bec787de0bf59cbf0cb684e8f40ee0ff1e93b7e340be9f0\": container with ID starting with 804a658e53d2d06c8bec787de0bf59cbf0cb684e8f40ee0ff1e93b7e340be9f0 not found: ID does not exist"
May 27 21:15:09.492482 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:09.492461    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"]
May 27 21:15:09.500684 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:09.500663    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-vtb52"]
May 27 21:15:09.933800 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:09.933762    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6611d9e0-cb9e-4b2f-a4b2-f29364327159" path="/var/lib/kubelet/pods/6611d9e0-cb9e-4b2f-a4b2-f29364327159/volumes"
May 27 21:15:13.812660 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:13.812633    2570 scope.go:117] "RemoveContainer" containerID="31983137d2d61a2308356d9762e7daca266186072b35e0add906f6b0fee36b87"
May 27 21:15:24.912511 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.912479    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"]
May 27 21:15:24.912977 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.912883    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6611d9e0-cb9e-4b2f-a4b2-f29364327159" containerName="manager"
May 27 21:15:24.912977 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.912901    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6611d9e0-cb9e-4b2f-a4b2-f29364327159" containerName="manager"
May 27 21:15:24.913082 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.912989    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="6611d9e0-cb9e-4b2f-a4b2-f29364327159" containerName="manager"
May 27 21:15:24.916677 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.916655    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:24.919985 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.919961    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vx6lh\""
May 27 21:15:24.931979 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.931951    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"]
May 27 21:15:24.983046 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.983001    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5987bd4-82cb-432c-a36f-f37f00caaaf2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:24.983046 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.983042    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5987bd4-82cb-432c-a36f-f37f00caaaf2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:24.983269 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.983066    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wkm8\" (UniqueName: \"kubernetes.io/projected/f5987bd4-82cb-432c-a36f-f37f00caaaf2-kube-api-access-2wkm8\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:24.983269 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.983119    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5987bd4-82cb-432c-a36f-f37f00caaaf2-workload-certs\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:24.983269 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.983165    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5987bd4-82cb-432c-a36f-f37f00caaaf2-istio-data\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:24.983414 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.983281    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987bd4-82cb-432c-a36f-f37f00caaaf2-workload-socket\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:24.983414 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.983336    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5987bd4-82cb-432c-a36f-f37f00caaaf2-istio-token\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:24.983414 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.983397    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987bd4-82cb-432c-a36f-f37f00caaaf2-credential-socket\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:24.983525 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:24.983431    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5987bd4-82cb-432c-a36f-f37f00caaaf2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.084500 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.084459    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5987bd4-82cb-432c-a36f-f37f00caaaf2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.084500 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.084504    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5987bd4-82cb-432c-a36f-f37f00caaaf2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.084748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.084524    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2wkm8\" (UniqueName: \"kubernetes.io/projected/f5987bd4-82cb-432c-a36f-f37f00caaaf2-kube-api-access-2wkm8\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.084748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.084541    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5987bd4-82cb-432c-a36f-f37f00caaaf2-workload-certs\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.084748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.084591    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5987bd4-82cb-432c-a36f-f37f00caaaf2-istio-data\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.084748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.084631    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987bd4-82cb-432c-a36f-f37f00caaaf2-workload-socket\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.084748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.084656    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5987bd4-82cb-432c-a36f-f37f00caaaf2-istio-token\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.084748 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.084747    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987bd4-82cb-432c-a36f-f37f00caaaf2-credential-socket\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.085049 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.085022    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5987bd4-82cb-432c-a36f-f37f00caaaf2-istio-data\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.085112 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.085046    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5987bd4-82cb-432c-a36f-f37f00caaaf2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.085281 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.085232    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987bd4-82cb-432c-a36f-f37f00caaaf2-workload-socket\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.085386 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.085303    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5987bd4-82cb-432c-a36f-f37f00caaaf2-workload-certs\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.085530 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.085510    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5987bd4-82cb-432c-a36f-f37f00caaaf2-credential-socket\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.085859 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.085839    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5987bd4-82cb-432c-a36f-f37f00caaaf2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.087074 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.087057    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5987bd4-82cb-432c-a36f-f37f00caaaf2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.087197 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.087180    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5987bd4-82cb-432c-a36f-f37f00caaaf2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.098732 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.098703    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5987bd4-82cb-432c-a36f-f37f00caaaf2-istio-token\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.099730 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.099713    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wkm8\" (UniqueName: \"kubernetes.io/projected/f5987bd4-82cb-432c-a36f-f37f00caaaf2-kube-api-access-2wkm8\") pod \"maas-default-gateway-openshift-default-7df95f575-p6wj6\" (UID: \"f5987bd4-82cb-432c-a36f-f37f00caaaf2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.231178 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.231093    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:25.367100 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.367068    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"]
May 27 21:15:25.370358 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:15:25.370329    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5987bd4_82cb_432c_a36f_f37f00caaaf2.slice/crio-a816f754e4402f2233d3edb7e6570f153d0fd8a9c6b3f9e165e6556bd8a466bc WatchSource:0}: Error finding container a816f754e4402f2233d3edb7e6570f153d0fd8a9c6b3f9e165e6556bd8a466bc: Status 404 returned error can't find the container with id a816f754e4402f2233d3edb7e6570f153d0fd8a9c6b3f9e165e6556bd8a466bc
May 27 21:15:25.520314 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:25.520224    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6" event={"ID":"f5987bd4-82cb-432c-a36f-f37f00caaaf2","Type":"ContainerStarted","Data":"a816f754e4402f2233d3edb7e6570f153d0fd8a9c6b3f9e165e6556bd8a466bc"}
May 27 21:15:28.544664 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:28.544628    2570 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 27 21:15:28.544907 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:28.544706    2570 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 27 21:15:28.544907 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:28.544739    2570 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
May 27 21:15:29.533411 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:29.533379    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6" event={"ID":"f5987bd4-82cb-432c-a36f-f37f00caaaf2","Type":"ContainerStarted","Data":"0742f554fa61fbc84ad506d2f6d637412af6ecd3fcbf432a53a3540810ac22ca"}
May 27 21:15:29.560058 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:29.560000    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6" podStartSLOduration=2.387923986 podStartE2EDuration="5.559986417s" podCreationTimestamp="2026-05-27 21:15:24 +0000 UTC" firstStartedPulling="2026-05-27 21:15:25.372320914 +0000 UTC m=+552.010946056" lastFinishedPulling="2026-05-27 21:15:28.544383337 +0000 UTC m=+555.183008487" observedRunningTime="2026-05-27 21:15:29.558448441 +0000 UTC m=+556.197073819" watchObservedRunningTime="2026-05-27 21:15:29.559986417 +0000 UTC m=+556.198611607"
May 27 21:15:30.231315 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.231275    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-2b2zx"]
May 27 21:15:30.234479 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.234454    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:30.234634 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.234552    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx"
May 27 21:15:30.236140 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.236121    2570 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:30.237308 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.237289    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
May 27 21:15:30.237623 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.237607    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-64wjq\""
May 27 21:15:30.245710 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.245688    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-2b2zx"]
May 27 21:15:30.326169 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.325690    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-2b2zx"]
May 27 21:15:30.330016 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.329985    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct4gp\" (UniqueName: \"kubernetes.io/projected/91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b-kube-api-access-ct4gp\") pod \"limitador-limitador-7d549b5b-2b2zx\" (UID: \"91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b\") " pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx"
May 27 21:15:30.330167 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.330029    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b-config-file\") pod \"limitador-limitador-7d549b5b-2b2zx\" (UID: \"91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b\") " pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx"
May 27 21:15:30.431347 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.431314    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ct4gp\" (UniqueName: \"kubernetes.io/projected/91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b-kube-api-access-ct4gp\") pod \"limitador-limitador-7d549b5b-2b2zx\" (UID: \"91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b\") " pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx"
May 27 21:15:30.431347 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.431353    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b-config-file\") pod \"limitador-limitador-7d549b5b-2b2zx\" (UID: \"91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b\") " pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx"
May 27 21:15:30.431954 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.431936    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b-config-file\") pod \"limitador-limitador-7d549b5b-2b2zx\" (UID: \"91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b\") " pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx"
May 27 21:15:30.440028 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.440001    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct4gp\" (UniqueName: \"kubernetes.io/projected/91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b-kube-api-access-ct4gp\") pod \"limitador-limitador-7d549b5b-2b2zx\" (UID: \"91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b\") " pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx"
May 27 21:15:30.536220 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.536141    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:30.537180 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.537157    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df95f575-p6wj6"
May 27 21:15:30.545130 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.545108    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx"
May 27 21:15:30.681626 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:30.681599    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-2b2zx"]
May 27 21:15:30.685272 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:15:30.685236    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91373e8b_d3ff_40c2_8ed3_dbe8e417cd8b.slice/crio-ba35880a41e8115e1511f5e5a79275701d145dd516e85d895939ca905bb58974 WatchSource:0}: Error finding container ba35880a41e8115e1511f5e5a79275701d145dd516e85d895939ca905bb58974: Status 404 returned error can't find the container with id ba35880a41e8115e1511f5e5a79275701d145dd516e85d895939ca905bb58974
May 27 21:15:31.541185 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:31.541145    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx" event={"ID":"91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b","Type":"ContainerStarted","Data":"ba35880a41e8115e1511f5e5a79275701d145dd516e85d895939ca905bb58974"}
May 27 21:15:33.549066 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:33.549031    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx" event={"ID":"91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b","Type":"ContainerStarted","Data":"4c076c216f7e342f28de56125c2adfc7a358050dd3c5af6ac3951d667589cc83"}
May 27 21:15:33.549438 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:33.549184    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx"
May 27 21:15:33.573904 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:33.573855    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx" podStartSLOduration=0.918843052 podStartE2EDuration="3.573843443s" podCreationTimestamp="2026-05-27 21:15:30 +0000 UTC" firstStartedPulling="2026-05-27 21:15:30.687185617 +0000 UTC m=+557.325810759" lastFinishedPulling="2026-05-27 21:15:33.342186002 +0000 UTC m=+559.980811150" observedRunningTime="2026-05-27 21:15:33.57199026 +0000 UTC m=+560.210615440" watchObservedRunningTime="2026-05-27 21:15:33.573843443 +0000 UTC m=+560.212468612"
May 27 21:15:44.553132 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:44.553093    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx"
May 27 21:15:47.291022 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:47.290992    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-2b2zx"]
May 27 21:15:47.291505 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:47.291210    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx" podUID="91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b" containerName="limitador" containerID="cri-o://4c076c216f7e342f28de56125c2adfc7a358050dd3c5af6ac3951d667589cc83" gracePeriod=30
May 27 21:15:47.823715 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:47.823692    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx"
May 27 21:15:47.980309 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:47.980278    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b-config-file\") pod \"91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b\" (UID: \"91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b\") "
May 27 21:15:47.980463 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:47.980322    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct4gp\" (UniqueName: \"kubernetes.io/projected/91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b-kube-api-access-ct4gp\") pod \"91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b\" (UID: \"91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b\") "
May 27 21:15:47.980651 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:47.980625    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b-config-file" (OuterVolumeSpecName: "config-file") pod "91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b" (UID: "91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 27 21:15:47.982294 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:47.982272    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b-kube-api-access-ct4gp" (OuterVolumeSpecName: "kube-api-access-ct4gp") pod "91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b" (UID: "91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b"). InnerVolumeSpecName "kube-api-access-ct4gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 21:15:48.081306 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:48.081271    2570 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b-config-file\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:15:48.081306 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:48.081302    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct4gp\" (UniqueName: \"kubernetes.io/projected/91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b-kube-api-access-ct4gp\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:15:48.595644 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:48.595609    2570 generic.go:358] "Generic (PLEG): container finished" podID="91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b" containerID="4c076c216f7e342f28de56125c2adfc7a358050dd3c5af6ac3951d667589cc83" exitCode=0
May 27 21:15:48.595644 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:48.595646    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx" event={"ID":"91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b","Type":"ContainerDied","Data":"4c076c216f7e342f28de56125c2adfc7a358050dd3c5af6ac3951d667589cc83"}
May 27 21:15:48.596151 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:48.595666    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx" event={"ID":"91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b","Type":"ContainerDied","Data":"ba35880a41e8115e1511f5e5a79275701d145dd516e85d895939ca905bb58974"}
May 27 21:15:48.596151 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:48.595681    2570 scope.go:117] "RemoveContainer" containerID="4c076c216f7e342f28de56125c2adfc7a358050dd3c5af6ac3951d667589cc83"
May 27 21:15:48.596151 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:48.595685    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7d549b5b-2b2zx"
May 27 21:15:48.603289 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:48.603267    2570 scope.go:117] "RemoveContainer" containerID="4c076c216f7e342f28de56125c2adfc7a358050dd3c5af6ac3951d667589cc83"
May 27 21:15:48.603529 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:15:48.603508    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c076c216f7e342f28de56125c2adfc7a358050dd3c5af6ac3951d667589cc83\": container with ID starting with 4c076c216f7e342f28de56125c2adfc7a358050dd3c5af6ac3951d667589cc83 not found: ID does not exist" containerID="4c076c216f7e342f28de56125c2adfc7a358050dd3c5af6ac3951d667589cc83"
May 27 21:15:48.603608 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:48.603540    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c076c216f7e342f28de56125c2adfc7a358050dd3c5af6ac3951d667589cc83"} err="failed to get container status \"4c076c216f7e342f28de56125c2adfc7a358050dd3c5af6ac3951d667589cc83\": rpc error: code = NotFound desc = could not find container \"4c076c216f7e342f28de56125c2adfc7a358050dd3c5af6ac3951d667589cc83\": container with ID starting with 4c076c216f7e342f28de56125c2adfc7a358050dd3c5af6ac3951d667589cc83 not found: ID does not exist"
May 27 21:15:48.622459 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:48.622434    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-2b2zx"]
May 27 21:15:48.626189 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:48.626166    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-2b2zx"]
May 27 21:15:49.938235 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:49.934403    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b" path="/var/lib/kubelet/pods/91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b/volumes"
May 27 21:15:51.195403 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.195371    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-68c4fbbd6f-7z5jw"]
May 27 21:15:51.195772 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.195697    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b" containerName="limitador"
May 27 21:15:51.195772 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.195710    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b" containerName="limitador"
May 27 21:15:51.195772 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.195763    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="91373e8b-d3ff-40c2-8ed3-dbe8e417cd8b" containerName="limitador"
May 27 21:15:51.199779 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.199759    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-68c4fbbd6f-7z5jw"
May 27 21:15:51.203242 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.203219    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rt8gh\""
May 27 21:15:51.203346 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.203255    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\""
May 27 21:15:51.219219 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.219193    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-68c4fbbd6f-7z5jw"]
May 27 21:15:51.309579 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.309532    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5kzc\" (UniqueName: \"kubernetes.io/projected/053a60a0-4a39-4a72-9073-e76728d10fb8-kube-api-access-f5kzc\") pod \"postgres-68c4fbbd6f-7z5jw\" (UID: \"053a60a0-4a39-4a72-9073-e76728d10fb8\") " pod="opendatahub/postgres-68c4fbbd6f-7z5jw"
May 27 21:15:51.309756 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.309692    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/053a60a0-4a39-4a72-9073-e76728d10fb8-data\") pod \"postgres-68c4fbbd6f-7z5jw\" (UID: \"053a60a0-4a39-4a72-9073-e76728d10fb8\") " pod="opendatahub/postgres-68c4fbbd6f-7z5jw"
May 27 21:15:51.410454 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.410420    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/053a60a0-4a39-4a72-9073-e76728d10fb8-data\") pod \"postgres-68c4fbbd6f-7z5jw\" (UID: \"053a60a0-4a39-4a72-9073-e76728d10fb8\") " pod="opendatahub/postgres-68c4fbbd6f-7z5jw"
May 27 21:15:51.410454 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.410461    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f5kzc\" (UniqueName: \"kubernetes.io/projected/053a60a0-4a39-4a72-9073-e76728d10fb8-kube-api-access-f5kzc\") pod \"postgres-68c4fbbd6f-7z5jw\" (UID: \"053a60a0-4a39-4a72-9073-e76728d10fb8\") " pod="opendatahub/postgres-68c4fbbd6f-7z5jw"
May 27 21:15:51.410888 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.410865    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/053a60a0-4a39-4a72-9073-e76728d10fb8-data\") pod \"postgres-68c4fbbd6f-7z5jw\" (UID: \"053a60a0-4a39-4a72-9073-e76728d10fb8\") " pod="opendatahub/postgres-68c4fbbd6f-7z5jw"
May 27 21:15:51.420903 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.420882    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5kzc\" (UniqueName: \"kubernetes.io/projected/053a60a0-4a39-4a72-9073-e76728d10fb8-kube-api-access-f5kzc\") pod \"postgres-68c4fbbd6f-7z5jw\" (UID: \"053a60a0-4a39-4a72-9073-e76728d10fb8\") " pod="opendatahub/postgres-68c4fbbd6f-7z5jw"
May 27 21:15:51.509392 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.509326    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-68c4fbbd6f-7z5jw"
May 27 21:15:51.633314 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:51.633284    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-68c4fbbd6f-7z5jw"]
May 27 21:15:51.636416 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:15:51.636391    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053a60a0_4a39_4a72_9073_e76728d10fb8.slice/crio-c4457a2f939bfe8677198979293a12fae0084e2d6b236f55a59ee2fd08394f85 WatchSource:0}: Error finding container c4457a2f939bfe8677198979293a12fae0084e2d6b236f55a59ee2fd08394f85: Status 404 returned error can't find the container with id c4457a2f939bfe8677198979293a12fae0084e2d6b236f55a59ee2fd08394f85
May 27 21:15:52.612207 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:52.612169    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-68c4fbbd6f-7z5jw" event={"ID":"053a60a0-4a39-4a72-9073-e76728d10fb8","Type":"ContainerStarted","Data":"c4457a2f939bfe8677198979293a12fae0084e2d6b236f55a59ee2fd08394f85"}
May 27 21:15:56.629277 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:56.629239    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-68c4fbbd6f-7z5jw" event={"ID":"053a60a0-4a39-4a72-9073-e76728d10fb8","Type":"ContainerStarted","Data":"abfdbcc282d198a99ab8fa8179ed42a51291ddce7dbd1ac51adc7857a9b697c0"}
May 27 21:15:56.629646 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:56.629356    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-68c4fbbd6f-7z5jw"
May 27 21:15:56.647993 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:15:56.647941    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/postgres-68c4fbbd6f-7z5jw" podStartSLOduration=1.17130597 podStartE2EDuration="5.647924202s" podCreationTimestamp="2026-05-27 21:15:51 +0000 UTC" firstStartedPulling="2026-05-27 21:15:51.637619162 +0000 UTC m=+578.276244304" lastFinishedPulling="2026-05-27 21:15:56.114237393 +0000 UTC m=+582.752862536" observedRunningTime="2026-05-27 21:15:56.647186897 +0000 UTC m=+583.285812057" watchObservedRunningTime="2026-05-27 21:15:56.647924202 +0000 UTC m=+583.286549367"
May 27 21:16:02.660628 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:02.660598    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-68c4fbbd6f-7z5jw"
May 27 21:16:42.092161 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:42.092079    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58b6c8fdc7-p65kb"]
May 27 21:16:42.095317 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:42.095298    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58b6c8fdc7-p65kb"
May 27 21:16:42.097837 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:42.097818    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
May 27 21:16:42.097955 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:42.097937    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vf558\""
May 27 21:16:42.108982 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:42.108953    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58b6c8fdc7-p65kb"]
May 27 21:16:42.211252 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:42.211215    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx6gf\" (UniqueName: \"kubernetes.io/projected/5a8e8c8d-a140-4d3e-991e-7b5000a3dd80-kube-api-access-gx6gf\") pod \"payload-processing-58b6c8fdc7-p65kb\" (UID: \"5a8e8c8d-a140-4d3e-991e-7b5000a3dd80\") " pod="openshift-ingress/payload-processing-58b6c8fdc7-p65kb"
May 27 21:16:42.312390 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:42.312354    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gx6gf\" (UniqueName: \"kubernetes.io/projected/5a8e8c8d-a140-4d3e-991e-7b5000a3dd80-kube-api-access-gx6gf\") pod \"payload-processing-58b6c8fdc7-p65kb\" (UID: \"5a8e8c8d-a140-4d3e-991e-7b5000a3dd80\") " pod="openshift-ingress/payload-processing-58b6c8fdc7-p65kb"
May 27 21:16:42.325326 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:42.325295    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx6gf\" (UniqueName: \"kubernetes.io/projected/5a8e8c8d-a140-4d3e-991e-7b5000a3dd80-kube-api-access-gx6gf\") pod \"payload-processing-58b6c8fdc7-p65kb\" (UID: \"5a8e8c8d-a140-4d3e-991e-7b5000a3dd80\") " pod="openshift-ingress/payload-processing-58b6c8fdc7-p65kb"
May 27 21:16:42.405878 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:42.405843    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58b6c8fdc7-p65kb"
May 27 21:16:42.532795 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:42.532773    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58b6c8fdc7-p65kb"]
May 27 21:16:42.534758 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:16:42.534729    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8e8c8d_a140_4d3e_991e_7b5000a3dd80.slice/crio-b995e9bb0c4f03e64ee0c328f071db1e78f71e6f311217f542c77f20f6d92d48 WatchSource:0}: Error finding container b995e9bb0c4f03e64ee0c328f071db1e78f71e6f311217f542c77f20f6d92d48: Status 404 returned error can't find the container with id b995e9bb0c4f03e64ee0c328f071db1e78f71e6f311217f542c77f20f6d92d48
May 27 21:16:42.776696 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:42.776604    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58b6c8fdc7-p65kb" event={"ID":"5a8e8c8d-a140-4d3e-991e-7b5000a3dd80","Type":"ContainerStarted","Data":"b995e9bb0c4f03e64ee0c328f071db1e78f71e6f311217f542c77f20f6d92d48"}
May 27 21:16:44.784664 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:44.784629    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58b6c8fdc7-p65kb" event={"ID":"5a8e8c8d-a140-4d3e-991e-7b5000a3dd80","Type":"ContainerStarted","Data":"e5e743b1e80a4b876cf096d4fe8c1281f5a1c3274c56e1ceb526f432a2fd0d80"}
May 27 21:16:44.785042 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:44.784747    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58b6c8fdc7-p65kb"
May 27 21:16:44.829295 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:44.829234    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58b6c8fdc7-p65kb" podStartSLOduration=1.07301935 podStartE2EDuration="2.82921887s" podCreationTimestamp="2026-05-27 21:16:42 +0000 UTC" firstStartedPulling="2026-05-27 21:16:42.536312088 +0000 UTC m=+629.174937234" lastFinishedPulling="2026-05-27 21:16:44.292511608 +0000 UTC m=+630.931136754" observedRunningTime="2026-05-27 21:16:44.82827818 +0000 UTC m=+631.466903345" watchObservedRunningTime="2026-05-27 21:16:44.82921887 +0000 UTC m=+631.467844031"
May 27 21:16:55.793044 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:16:55.793012    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58b6c8fdc7-p65kb"
May 27 21:17:13.841236 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:13.841207    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"]
May 27 21:17:13.844651 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:13.844631    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:13.848120 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:13.848089    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
May 27 21:17:13.848120 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:13.848116    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
May 27 21:17:13.848268 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:13.848150    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
May 27 21:17:13.848268 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:13.848094    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-knrtt\""
May 27 21:17:13.854706 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:13.854683    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"]
May 27 21:17:13.952026 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:13.952001    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3844-54a1-4470-9c2c-3268a6abf70e-kserve-provision-location\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:13.952172 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:13.952047    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cd3844-54a1-4470-9c2c-3268a6abf70e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:13.952172 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:13.952098    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3844-54a1-4470-9c2c-3268a6abf70e-home\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:13.952172 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:13.952120    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3844-54a1-4470-9c2c-3268a6abf70e-dshm\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:13.952172 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:13.952145    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7tms\" (UniqueName: \"kubernetes.io/projected/a8cd3844-54a1-4470-9c2c-3268a6abf70e-kube-api-access-t7tms\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:13.952293 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:13.952189    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3844-54a1-4470-9c2c-3268a6abf70e-model-cache\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:14.052706 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:14.052672    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cd3844-54a1-4470-9c2c-3268a6abf70e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:14.052706 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:14.052712    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3844-54a1-4470-9c2c-3268a6abf70e-home\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:14.052990 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:14.052729    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3844-54a1-4470-9c2c-3268a6abf70e-dshm\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:14.052990 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:14.052749    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t7tms\" (UniqueName: \"kubernetes.io/projected/a8cd3844-54a1-4470-9c2c-3268a6abf70e-kube-api-access-t7tms\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:14.052990 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:14.052773    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3844-54a1-4470-9c2c-3268a6abf70e-model-cache\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:14.052990 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:14.052821    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3844-54a1-4470-9c2c-3268a6abf70e-kserve-provision-location\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:14.053200 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:14.053098    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3844-54a1-4470-9c2c-3268a6abf70e-home\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:14.053200 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:14.053183    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3844-54a1-4470-9c2c-3268a6abf70e-kserve-provision-location\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:14.053281 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:14.053258    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3844-54a1-4470-9c2c-3268a6abf70e-model-cache\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:14.055052 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:14.055024    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cd3844-54a1-4470-9c2c-3268a6abf70e-dshm\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:14.055398 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:14.055379    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cd3844-54a1-4470-9c2c-3268a6abf70e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:14.060649 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:14.060629    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7tms\" (UniqueName: \"kubernetes.io/projected/a8cd3844-54a1-4470-9c2c-3268a6abf70e-kube-api-access-t7tms\") pod \"e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h\" (UID: \"a8cd3844-54a1-4470-9c2c-3268a6abf70e\") " pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:14.154929 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:14.154896    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:14.287066 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:14.286990    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"]
May 27 21:17:14.289589 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:17:14.289544    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cd3844_54a1_4470_9c2c_3268a6abf70e.slice/crio-3e1ed2ace5f4ba5352c2393c7334ebe801baa2f581178d7490c978bb5dec9561 WatchSource:0}: Error finding container 3e1ed2ace5f4ba5352c2393c7334ebe801baa2f581178d7490c978bb5dec9561: Status 404 returned error can't find the container with id 3e1ed2ace5f4ba5352c2393c7334ebe801baa2f581178d7490c978bb5dec9561
May 27 21:17:14.880028 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:14.879991    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h" event={"ID":"a8cd3844-54a1-4470-9c2c-3268a6abf70e","Type":"ContainerStarted","Data":"3e1ed2ace5f4ba5352c2393c7334ebe801baa2f581178d7490c978bb5dec9561"}
May 27 21:17:21.907316 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:21.907261    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h" event={"ID":"a8cd3844-54a1-4470-9c2c-3268a6abf70e","Type":"ContainerStarted","Data":"d72b450495e499fe42f507803f3e4156a90b0ae698b382940b597a9feef595a7"}
May 27 21:17:26.924219 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:26.924186    2570 generic.go:358] "Generic (PLEG): container finished" podID="a8cd3844-54a1-4470-9c2c-3268a6abf70e" containerID="d72b450495e499fe42f507803f3e4156a90b0ae698b382940b597a9feef595a7" exitCode=0
May 27 21:17:26.924658 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:26.924254    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h" event={"ID":"a8cd3844-54a1-4470-9c2c-3268a6abf70e","Type":"ContainerDied","Data":"d72b450495e499fe42f507803f3e4156a90b0ae698b382940b597a9feef595a7"}
May 27 21:17:26.924843 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:26.924830    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 27 21:17:28.932656 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:28.932622    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h" event={"ID":"a8cd3844-54a1-4470-9c2c-3268a6abf70e","Type":"ContainerStarted","Data":"df7076c7ea316783b0c764f71fd6f495f9b6804fc334e9fba2d6647e2e80e317"}
May 27 21:17:28.933024 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:28.932819    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:28.954992 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:28.954943    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h" podStartSLOduration=2.280221239 podStartE2EDuration="15.954928784s" podCreationTimestamp="2026-05-27 21:17:13 +0000 UTC" firstStartedPulling="2026-05-27 21:17:14.291184499 +0000 UTC m=+660.929809641" lastFinishedPulling="2026-05-27 21:17:27.96589204 +0000 UTC m=+674.604517186" observedRunningTime="2026-05-27 21:17:28.952479077 +0000 UTC m=+675.591104275" watchObservedRunningTime="2026-05-27 21:17:28.954928784 +0000 UTC m=+675.593553947"
May 27 21:17:36.356116 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.356082    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"]
May 27 21:17:36.363300 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.363277    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.366962 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.366941    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
May 27 21:17:36.377442 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.377420    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"]
May 27 21:17:36.459257 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.459221    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.459257 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.459260    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-kserve-provision-location\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.459462 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.459289    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.459462 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.459311    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.459462 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.459348    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.459462 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.459392    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gvp5\" (UniqueName: \"kubernetes.io/projected/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-kube-api-access-6gvp5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.560708 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.560669    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.560900 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.560728    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-kserve-provision-location\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.560900 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.560760    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.560900 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.560777    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.560900 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.560806    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.560900 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.560851    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6gvp5\" (UniqueName: \"kubernetes.io/projected/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-kube-api-access-6gvp5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.561319 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.561293    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.561396 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.561319    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.561396 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.561336    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-kserve-provision-location\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.563030 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.563011    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.563552 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.563528    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.570027 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.570002    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gvp5\" (UniqueName: \"kubernetes.io/projected/cc4a497d-8c9a-40f6-9f6c-3ee788c67ead-kube-api-access-6gvp5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2\" (UID: \"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.673516 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.673480    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:36.805472 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.805436    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"]
May 27 21:17:36.808802 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:17:36.808776    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4a497d_8c9a_40f6_9f6c_3ee788c67ead.slice/crio-6165c5a483ac1c6121fa36786b1138e01b097514fffa56dfb1d753f76898c9b8 WatchSource:0}: Error finding container 6165c5a483ac1c6121fa36786b1138e01b097514fffa56dfb1d753f76898c9b8: Status 404 returned error can't find the container with id 6165c5a483ac1c6121fa36786b1138e01b097514fffa56dfb1d753f76898c9b8
May 27 21:17:36.958020 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.957932    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2" event={"ID":"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead","Type":"ContainerStarted","Data":"7e55a496593a25ee844114c7bbcd519a79d444bcea6186d0ba61296a45498f9a"}
May 27 21:17:36.958020 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:36.957976    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2" event={"ID":"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead","Type":"ContainerStarted","Data":"6165c5a483ac1c6121fa36786b1138e01b097514fffa56dfb1d753f76898c9b8"}
May 27 21:17:39.948669 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:39.948638    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h"
May 27 21:17:42.978884 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:42.978848    2570 generic.go:358] "Generic (PLEG): container finished" podID="cc4a497d-8c9a-40f6-9f6c-3ee788c67ead" containerID="7e55a496593a25ee844114c7bbcd519a79d444bcea6186d0ba61296a45498f9a" exitCode=0
May 27 21:17:42.979282 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:42.978924    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2" event={"ID":"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead","Type":"ContainerDied","Data":"7e55a496593a25ee844114c7bbcd519a79d444bcea6186d0ba61296a45498f9a"}
May 27 21:17:43.982710 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:43.982672    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2" event={"ID":"cc4a497d-8c9a-40f6-9f6c-3ee788c67ead","Type":"ContainerStarted","Data":"4f6e8422954423ba3f1981c39f8e6536bf35ff02f7434d45a5532a434c150b89"}
May 27 21:17:43.983182 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:43.982918    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:17:44.005002 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:44.004949    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2" podStartSLOduration=7.826965416 podStartE2EDuration="8.004936169s" podCreationTimestamp="2026-05-27 21:17:36 +0000 UTC" firstStartedPulling="2026-05-27 21:17:42.979502813 +0000 UTC m=+689.618127955" lastFinishedPulling="2026-05-27 21:17:43.157473563 +0000 UTC m=+689.796098708" observedRunningTime="2026-05-27 21:17:44.00338301 +0000 UTC m=+690.642008175" watchObservedRunningTime="2026-05-27 21:17:44.004936169 +0000 UTC m=+690.643561333"
May 27 21:17:47.070851 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.070811    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"]
May 27 21:17:47.075119 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.075096    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.078274 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.078252    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
May 27 21:17:47.083922 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.083898    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"]
May 27 21:17:47.250140 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.250098    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e4669b0-55c8-4296-852c-5f305fbf8717-model-cache\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.250324 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.250152    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e4669b0-55c8-4296-852c-5f305fbf8717-home\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.250324 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.250214    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e4669b0-55c8-4296-852c-5f305fbf8717-dshm\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.250324 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.250240    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9f6h\" (UniqueName: \"kubernetes.io/projected/4e4669b0-55c8-4296-852c-5f305fbf8717-kube-api-access-l9f6h\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.250445 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.250321    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e4669b0-55c8-4296-852c-5f305fbf8717-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.250445 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.250361    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e4669b0-55c8-4296-852c-5f305fbf8717-kserve-provision-location\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.351772 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.351691    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e4669b0-55c8-4296-852c-5f305fbf8717-home\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.351772 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.351758    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e4669b0-55c8-4296-852c-5f305fbf8717-dshm\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.351982 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.351783    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l9f6h\" (UniqueName: \"kubernetes.io/projected/4e4669b0-55c8-4296-852c-5f305fbf8717-kube-api-access-l9f6h\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.351982 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.351820    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e4669b0-55c8-4296-852c-5f305fbf8717-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.351982 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.351950    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e4669b0-55c8-4296-852c-5f305fbf8717-kserve-provision-location\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.352124 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.352048    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e4669b0-55c8-4296-852c-5f305fbf8717-model-cache\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.352170 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.352120    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e4669b0-55c8-4296-852c-5f305fbf8717-home\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.352308 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.352284    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e4669b0-55c8-4296-852c-5f305fbf8717-kserve-provision-location\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.352387 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.352372    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e4669b0-55c8-4296-852c-5f305fbf8717-model-cache\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.353987 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.353960    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e4669b0-55c8-4296-852c-5f305fbf8717-dshm\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.354233 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.354217    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e4669b0-55c8-4296-852c-5f305fbf8717-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.359819 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.359798    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9f6h\" (UniqueName: \"kubernetes.io/projected/4e4669b0-55c8-4296-852c-5f305fbf8717-kube-api-access-l9f6h\") pod \"e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28\" (UID: \"4e4669b0-55c8-4296-852c-5f305fbf8717\") " pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.385259 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.385241    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:47.525138 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.525106    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"]
May 27 21:17:47.528150 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:17:47.528120    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4669b0_55c8_4296_852c_5f305fbf8717.slice/crio-f99bb901ddbaafdbf932698e8516823aa47d0261153361efa735a05962b165b6 WatchSource:0}: Error finding container f99bb901ddbaafdbf932698e8516823aa47d0261153361efa735a05962b165b6: Status 404 returned error can't find the container with id f99bb901ddbaafdbf932698e8516823aa47d0261153361efa735a05962b165b6
May 27 21:17:47.997020 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.996986    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28" event={"ID":"4e4669b0-55c8-4296-852c-5f305fbf8717","Type":"ContainerStarted","Data":"29531c649a7931d1e3feb9c6531cc69fbcea9e2acb1ca4b221f151d52b64e359"}
May 27 21:17:47.997020 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:47.997024    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28" event={"ID":"4e4669b0-55c8-4296-852c-5f305fbf8717","Type":"ContainerStarted","Data":"f99bb901ddbaafdbf932698e8516823aa47d0261153361efa735a05962b165b6"}
May 27 21:17:53.016053 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:53.016020    2570 generic.go:358] "Generic (PLEG): container finished" podID="4e4669b0-55c8-4296-852c-5f305fbf8717" containerID="29531c649a7931d1e3feb9c6531cc69fbcea9e2acb1ca4b221f151d52b64e359" exitCode=0
May 27 21:17:53.016397 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:53.016076    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28" event={"ID":"4e4669b0-55c8-4296-852c-5f305fbf8717","Type":"ContainerDied","Data":"29531c649a7931d1e3feb9c6531cc69fbcea9e2acb1ca4b221f151d52b64e359"}
May 27 21:17:54.020126 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:54.020094    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28" event={"ID":"4e4669b0-55c8-4296-852c-5f305fbf8717","Type":"ContainerStarted","Data":"52fd5904067cd2e00b3abeff60d5783294097159c67fd9329970ada70608adf0"}
May 27 21:17:54.020477 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:54.020288    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:17:54.040747 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:54.040705    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28" podStartSLOduration=6.845960696 podStartE2EDuration="7.040693409s" podCreationTimestamp="2026-05-27 21:17:47 +0000 UTC" firstStartedPulling="2026-05-27 21:17:53.016733993 +0000 UTC m=+699.655359136" lastFinishedPulling="2026-05-27 21:17:53.211466703 +0000 UTC m=+699.850091849" observedRunningTime="2026-05-27 21:17:54.039324162 +0000 UTC m=+700.677949329" watchObservedRunningTime="2026-05-27 21:17:54.040693409 +0000 UTC m=+700.679318573"
May 27 21:17:55.001525 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:17:55.001497    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2"
May 27 21:18:05.036682 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:18:05.036651    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28"
May 27 21:30:18.080126 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.080093    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"]
May 27 21:30:18.080624 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.080328    2570 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57" podUID="02436455-741f-4d29-ae2e-dcedad32bd18" containerName="manager" containerID="cri-o://41f300203a3a3e6b39065e985606b60a065a86df460dce7ef83d5c1a177e008d" gracePeriod=10
May 27 21:30:18.332140 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.332088    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"
May 27 21:30:18.490213 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.490179    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z54x\" (UniqueName: \"kubernetes.io/projected/02436455-741f-4d29-ae2e-dcedad32bd18-kube-api-access-8z54x\") pod \"02436455-741f-4d29-ae2e-dcedad32bd18\" (UID: \"02436455-741f-4d29-ae2e-dcedad32bd18\") "
May 27 21:30:18.490405 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.490243    2570 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02436455-741f-4d29-ae2e-dcedad32bd18-extensions-socket-volume\") pod \"02436455-741f-4d29-ae2e-dcedad32bd18\" (UID: \"02436455-741f-4d29-ae2e-dcedad32bd18\") "
May 27 21:30:18.490638 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.490615    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02436455-741f-4d29-ae2e-dcedad32bd18-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "02436455-741f-4d29-ae2e-dcedad32bd18" (UID: "02436455-741f-4d29-ae2e-dcedad32bd18"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 27 21:30:18.492244 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.492221    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02436455-741f-4d29-ae2e-dcedad32bd18-kube-api-access-8z54x" (OuterVolumeSpecName: "kube-api-access-8z54x") pod "02436455-741f-4d29-ae2e-dcedad32bd18" (UID: "02436455-741f-4d29-ae2e-dcedad32bd18"). InnerVolumeSpecName "kube-api-access-8z54x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 27 21:30:18.506424 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.506398    2570 generic.go:358] "Generic (PLEG): container finished" podID="02436455-741f-4d29-ae2e-dcedad32bd18" containerID="41f300203a3a3e6b39065e985606b60a065a86df460dce7ef83d5c1a177e008d" exitCode=0
May 27 21:30:18.506522 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.506457    2570 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"
May 27 21:30:18.506522 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.506474    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57" event={"ID":"02436455-741f-4d29-ae2e-dcedad32bd18","Type":"ContainerDied","Data":"41f300203a3a3e6b39065e985606b60a065a86df460dce7ef83d5c1a177e008d"}
May 27 21:30:18.506522 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.506518    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57" event={"ID":"02436455-741f-4d29-ae2e-dcedad32bd18","Type":"ContainerDied","Data":"f23f550f967a7423957e884c54d8343609bd7ddf9f63eaa980d53100a75d1977"}
May 27 21:30:18.506704 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.506539    2570 scope.go:117] "RemoveContainer" containerID="41f300203a3a3e6b39065e985606b60a065a86df460dce7ef83d5c1a177e008d"
May 27 21:30:18.514797 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.514778    2570 scope.go:117] "RemoveContainer" containerID="41f300203a3a3e6b39065e985606b60a065a86df460dce7ef83d5c1a177e008d"
May 27 21:30:18.515024 ip-10-0-128-93 kubenswrapper[2570]: E0527 21:30:18.515006    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f300203a3a3e6b39065e985606b60a065a86df460dce7ef83d5c1a177e008d\": container with ID starting with 41f300203a3a3e6b39065e985606b60a065a86df460dce7ef83d5c1a177e008d not found: ID does not exist" containerID="41f300203a3a3e6b39065e985606b60a065a86df460dce7ef83d5c1a177e008d"
May 27 21:30:18.515090 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.515036    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f300203a3a3e6b39065e985606b60a065a86df460dce7ef83d5c1a177e008d"} err="failed to get container status \"41f300203a3a3e6b39065e985606b60a065a86df460dce7ef83d5c1a177e008d\": rpc error: code = NotFound desc = could not find container \"41f300203a3a3e6b39065e985606b60a065a86df460dce7ef83d5c1a177e008d\": container with ID starting with 41f300203a3a3e6b39065e985606b60a065a86df460dce7ef83d5c1a177e008d not found: ID does not exist"
May 27 21:30:18.528006 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.527984    2570 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"]
May 27 21:30:18.531335 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.531308    2570 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-s4p57"]
May 27 21:30:18.591619 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.591511    2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/02436455-741f-4d29-ae2e-dcedad32bd18-extensions-socket-volume\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:30:18.591619 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:18.591544    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z54x\" (UniqueName: \"kubernetes.io/projected/02436455-741f-4d29-ae2e-dcedad32bd18-kube-api-access-8z54x\") on node \"ip-10-0-128-93.ec2.internal\" DevicePath \"\""
May 27 21:30:19.933899 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:30:19.933867    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02436455-741f-4d29-ae2e-dcedad32bd18" path="/var/lib/kubelet/pods/02436455-741f-4d29-ae2e-dcedad32bd18/volumes"
May 27 21:31:24.318233 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.318202    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s"]
May 27 21:31:24.318746 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.318569    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02436455-741f-4d29-ae2e-dcedad32bd18" containerName="manager"
May 27 21:31:24.318746 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.318582    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="02436455-741f-4d29-ae2e-dcedad32bd18" containerName="manager"
May 27 21:31:24.318746 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.318654    2570 memory_manager.go:356] "RemoveStaleState removing state" podUID="02436455-741f-4d29-ae2e-dcedad32bd18" containerName="manager"
May 27 21:31:24.321851 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.321836    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s"
May 27 21:31:24.327049 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.327025    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 27 21:31:24.327261 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.327248    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l7wxj\""
May 27 21:31:24.327592 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.327571    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 27 21:31:24.346062 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.346033    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s"]
May 27 21:31:24.445350 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.445316    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9962a2b-d91e-4a0a-b6eb-332c156fe97f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55c7f4c975-6ss2s\" (UID: \"b9962a2b-d91e-4a0a-b6eb-332c156fe97f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s"
May 27 21:31:24.445513 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.445417    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b46f\" (UniqueName: \"kubernetes.io/projected/b9962a2b-d91e-4a0a-b6eb-332c156fe97f-kube-api-access-5b46f\") pod \"kuadrant-operator-controller-manager-55c7f4c975-6ss2s\" (UID: \"b9962a2b-d91e-4a0a-b6eb-332c156fe97f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s"
May 27 21:31:24.546703 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.546654    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5b46f\" (UniqueName: \"kubernetes.io/projected/b9962a2b-d91e-4a0a-b6eb-332c156fe97f-kube-api-access-5b46f\") pod \"kuadrant-operator-controller-manager-55c7f4c975-6ss2s\" (UID: \"b9962a2b-d91e-4a0a-b6eb-332c156fe97f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s"
May 27 21:31:24.546703 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.546717    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9962a2b-d91e-4a0a-b6eb-332c156fe97f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55c7f4c975-6ss2s\" (UID: \"b9962a2b-d91e-4a0a-b6eb-332c156fe97f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s"
May 27 21:31:24.547048 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.547032    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9962a2b-d91e-4a0a-b6eb-332c156fe97f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55c7f4c975-6ss2s\" (UID: \"b9962a2b-d91e-4a0a-b6eb-332c156fe97f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s"
May 27 21:31:24.563299 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.563269    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b46f\" (UniqueName: \"kubernetes.io/projected/b9962a2b-d91e-4a0a-b6eb-332c156fe97f-kube-api-access-5b46f\") pod \"kuadrant-operator-controller-manager-55c7f4c975-6ss2s\" (UID: \"b9962a2b-d91e-4a0a-b6eb-332c156fe97f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s"
May 27 21:31:24.632067 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.632036    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s"
May 27 21:31:24.769737 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.769705    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s"]
May 27 21:31:24.773067 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:31:24.773042    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9962a2b_d91e_4a0a_b6eb_332c156fe97f.slice/crio-2dc4e3495f62f14aded1bec28c32bb78f153f2cff370c74c8b01aef02a105044 WatchSource:0}: Error finding container 2dc4e3495f62f14aded1bec28c32bb78f153f2cff370c74c8b01aef02a105044: Status 404 returned error can't find the container with id 2dc4e3495f62f14aded1bec28c32bb78f153f2cff370c74c8b01aef02a105044
May 27 21:31:24.775494 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:24.775476    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 27 21:31:25.737504 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:25.737468    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s" event={"ID":"b9962a2b-d91e-4a0a-b6eb-332c156fe97f","Type":"ContainerStarted","Data":"e65633495cfdb5c06b844d8da88c710f02f9c514763c524a268131f51e956437"}
May 27 21:31:25.737885 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:25.737512    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s" event={"ID":"b9962a2b-d91e-4a0a-b6eb-332c156fe97f","Type":"ContainerStarted","Data":"2dc4e3495f62f14aded1bec28c32bb78f153f2cff370c74c8b01aef02a105044"}
May 27 21:31:25.737885 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:25.737540    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s"
May 27 21:31:25.784199 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:25.784157    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s" podStartSLOduration=1.784143558 podStartE2EDuration="1.784143558s" podCreationTimestamp="2026-05-27 21:31:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 21:31:25.781478902 +0000 UTC m=+1512.420104065" watchObservedRunningTime="2026-05-27 21:31:25.784143558 +0000 UTC m=+1512.422768720"
May 27 21:31:36.744055 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:31:36.743984    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55c7f4c975-6ss2s"
May 27 21:41:06.565090 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:06.565055    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-856948b99f-bjkjp_7037f41c-22ab-4a51-888d-40b2c8998c25/manager/0.log"
May 27 21:41:07.045981 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:07.045952    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-858dbf95b8-2jzbs_40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7/manager/2.log"
May 27 21:41:07.273072 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:07.273029    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6678cd66dc-mk4qm_835def55-9d49-48a1-8c71-3755952cda77/manager/0.log"
May 27 21:41:07.492780 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:07.492749    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-68c4fbbd6f-7z5jw_053a60a0-4a39-4a72-9073-e76728d10fb8/postgres/0.log"
May 27 21:41:09.277033 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:09.277004    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-55c7f4c975-6ss2s_b9962a2b-d91e-4a0a-b6eb-332c156fe97f/manager/0.log"
May 27 21:41:09.950162 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:09.950139    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-798958bb55-jnxvm_73350e66-fa44-49e6-aa5f-e02023fa01be/discovery/0.log"
May 27 21:41:10.055205 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:10.055180    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f6f4b4c58-84r4f_2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae/kube-auth-proxy/0.log"
May 27 21:41:10.277140 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:10.277064    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df95f575-p6wj6_f5987bd4-82cb-432c-a36f-f37f00caaaf2/istio-proxy/0.log"
May 27 21:41:10.393465 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:10.393431    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58b6c8fdc7-p65kb_5a8e8c8d-a140-4d3e-991e-7b5000a3dd80/payload-processing/0.log"
May 27 21:41:10.819255 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:10.819228    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28_4e4669b0-55c8-4296-852c-5f305fbf8717/storage-initializer/0.log"
May 27 21:41:10.825845 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:10.825825    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-7f849f6b56-tcg28_4e4669b0-55c8-4296-852c-5f305fbf8717/main/0.log"
May 27 21:41:10.936608 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:10.936576    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h_a8cd3844-54a1-4470-9c2c-3268a6abf70e/storage-initializer/0.log"
May 27 21:41:10.944800 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:10.944762    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7bb4cdb4d7-gdm6h_a8cd3844-54a1-4470-9c2c-3268a6abf70e/main/0.log"
May 27 21:41:11.170573 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:11.170540    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2_cc4a497d-8c9a-40f6-9f6c-3ee788c67ead/storage-initializer/0.log"
May 27 21:41:11.177646 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:11.177625    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccknlj2_cc4a497d-8c9a-40f6-9f6c-3ee788c67ead/main/0.log"
May 27 21:41:18.633394 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:18.633362    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5ws56_0216b723-92f1-4f86-a216-384ab8ea7891/global-pull-secret-syncer/0.log"
May 27 21:41:18.884910 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:18.884833    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cwbhz_5e3d6151-d7e1-4794-b820-4cf1bd1c12d1/konnectivity-agent/0.log"
May 27 21:41:18.905268 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:18.905243    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-93.ec2.internal_96ba32301ea110b32229358267c0d072/haproxy/0.log"
May 27 21:41:22.967847 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:22.967804    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-55c7f4c975-6ss2s_b9962a2b-d91e-4a0a-b6eb-332c156fe97f/manager/0.log"
May 27 21:41:25.096958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:25.096931    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l2jgs_9a4d1aa1-8313-44ae-98c0-343b8cacd00b/node-exporter/0.log"
May 27 21:41:25.122668 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:25.122633    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l2jgs_9a4d1aa1-8313-44ae-98c0-343b8cacd00b/kube-rbac-proxy/0.log"
May 27 21:41:25.149743 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:25.149720    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l2jgs_9a4d1aa1-8313-44ae-98c0-343b8cacd00b/init-textfile/0.log"
May 27 21:41:26.823952 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:26.823926    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-wzb2c_3c91cead-d228-41c8-be05-0b404e6f71d6/networking-console-plugin/0.log"
May 27 21:41:27.376013 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.375981    2570 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"]
May 27 21:41:27.378473 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.378451    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.381067 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.381047    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t5vz8\"/\"kube-root-ca.crt\""
May 27 21:41:27.381175 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.381113    2570 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t5vz8\"/\"openshift-service-ca.crt\""
May 27 21:41:27.381960 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.381947    2570 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t5vz8\"/\"default-dockercfg-qz47r\""
May 27 21:41:27.388890 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.388860    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"]
May 27 21:41:27.460077 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.460050    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a5849b6-458f-485e-b47d-616b8c8358a8-sys\") pod \"perf-node-gather-daemonset-72m69\" (UID: \"6a5849b6-458f-485e-b47d-616b8c8358a8\") " pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.460242 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.460085    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a5849b6-458f-485e-b47d-616b8c8358a8-proc\") pod \"perf-node-gather-daemonset-72m69\" (UID: \"6a5849b6-458f-485e-b47d-616b8c8358a8\") " pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.460242 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.460112    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a5849b6-458f-485e-b47d-616b8c8358a8-lib-modules\") pod \"perf-node-gather-daemonset-72m69\" (UID: \"6a5849b6-458f-485e-b47d-616b8c8358a8\") " pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.460242 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.460138    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxw98\" (UniqueName: \"kubernetes.io/projected/6a5849b6-458f-485e-b47d-616b8c8358a8-kube-api-access-fxw98\") pod \"perf-node-gather-daemonset-72m69\" (UID: \"6a5849b6-458f-485e-b47d-616b8c8358a8\") " pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.460242 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.460166    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a5849b6-458f-485e-b47d-616b8c8358a8-podres\") pod \"perf-node-gather-daemonset-72m69\" (UID: \"6a5849b6-458f-485e-b47d-616b8c8358a8\") " pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.561347 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.561313    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a5849b6-458f-485e-b47d-616b8c8358a8-sys\") pod \"perf-node-gather-daemonset-72m69\" (UID: \"6a5849b6-458f-485e-b47d-616b8c8358a8\") " pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.561347 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.561349    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a5849b6-458f-485e-b47d-616b8c8358a8-proc\") pod \"perf-node-gather-daemonset-72m69\" (UID: \"6a5849b6-458f-485e-b47d-616b8c8358a8\") " pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.561582 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.561368    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a5849b6-458f-485e-b47d-616b8c8358a8-lib-modules\") pod \"perf-node-gather-daemonset-72m69\" (UID: \"6a5849b6-458f-485e-b47d-616b8c8358a8\") " pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.561582 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.561385    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fxw98\" (UniqueName: \"kubernetes.io/projected/6a5849b6-458f-485e-b47d-616b8c8358a8-kube-api-access-fxw98\") pod \"perf-node-gather-daemonset-72m69\" (UID: \"6a5849b6-458f-485e-b47d-616b8c8358a8\") " pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.561582 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.561414    2570 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a5849b6-458f-485e-b47d-616b8c8358a8-podres\") pod \"perf-node-gather-daemonset-72m69\" (UID: \"6a5849b6-458f-485e-b47d-616b8c8358a8\") " pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.561582 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.561458    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a5849b6-458f-485e-b47d-616b8c8358a8-proc\") pod \"perf-node-gather-daemonset-72m69\" (UID: \"6a5849b6-458f-485e-b47d-616b8c8358a8\") " pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.561582 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.561461    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a5849b6-458f-485e-b47d-616b8c8358a8-sys\") pod \"perf-node-gather-daemonset-72m69\" (UID: \"6a5849b6-458f-485e-b47d-616b8c8358a8\") " pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.561582 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.561530    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a5849b6-458f-485e-b47d-616b8c8358a8-podres\") pod \"perf-node-gather-daemonset-72m69\" (UID: \"6a5849b6-458f-485e-b47d-616b8c8358a8\") " pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.561582 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.561536    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a5849b6-458f-485e-b47d-616b8c8358a8-lib-modules\") pod \"perf-node-gather-daemonset-72m69\" (UID: \"6a5849b6-458f-485e-b47d-616b8c8358a8\") " pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.569701 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.569682    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxw98\" (UniqueName: \"kubernetes.io/projected/6a5849b6-458f-485e-b47d-616b8c8358a8-kube-api-access-fxw98\") pod \"perf-node-gather-daemonset-72m69\" (UID: \"6a5849b6-458f-485e-b47d-616b8c8358a8\") " pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.688410 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.688344    2570 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:27.809047 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.809023    2570 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"]
May 27 21:41:27.811546 ip-10-0-128-93 kubenswrapper[2570]: W0527 21:41:27.811520    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6a5849b6_458f_485e_b47d_616b8c8358a8.slice/crio-015e505ab5f05838d4af8cbe6fc23a074044b341de4a90dbe8f0d67e65cb6c70 WatchSource:0}: Error finding container 015e505ab5f05838d4af8cbe6fc23a074044b341de4a90dbe8f0d67e65cb6c70: Status 404 returned error can't find the container with id 015e505ab5f05838d4af8cbe6fc23a074044b341de4a90dbe8f0d67e65cb6c70
May 27 21:41:27.813002 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:27.812986    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 27 21:41:28.695683 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:28.695648    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69" event={"ID":"6a5849b6-458f-485e-b47d-616b8c8358a8","Type":"ContainerStarted","Data":"6340569b5f7b7257dc2158fc253bfb61b81d83c8cd80af1a0c5c7fb9d4c665da"}
May 27 21:41:28.695683 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:28.695685    2570 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69" event={"ID":"6a5849b6-458f-485e-b47d-616b8c8358a8","Type":"ContainerStarted","Data":"015e505ab5f05838d4af8cbe6fc23a074044b341de4a90dbe8f0d67e65cb6c70"}
May 27 21:41:28.696144 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:28.695785    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:28.713276 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:28.713235    2570 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69" podStartSLOduration=1.713224037 podStartE2EDuration="1.713224037s" podCreationTimestamp="2026-05-27 21:41:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-27 21:41:28.712037843 +0000 UTC m=+2115.350663008" watchObservedRunningTime="2026-05-27 21:41:28.713224037 +0000 UTC m=+2115.351849199"
May 27 21:41:29.304632 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:29.304603    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4d9hh_41d38e9c-e685-40bc-a1fc-74d393746458/dns/0.log"
May 27 21:41:29.326311 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:29.326283    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4d9hh_41d38e9c-e685-40bc-a1fc-74d393746458/kube-rbac-proxy/0.log"
May 27 21:41:29.441229 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:29.441202    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7cjkr_a31a703f-7353-451c-9112-accce453fe8c/dns-node-resolver/0.log"
May 27 21:41:30.006411 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:30.006386    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ktm2q_8627c382-6754-44fb-bb08-de5c1cd78407/node-ca/0.log"
May 27 21:41:31.037087 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:31.037058    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-798958bb55-jnxvm_73350e66-fa44-49e6-aa5f-e02023fa01be/discovery/0.log"
May 27 21:41:31.057706 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:31.057684    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f6f4b4c58-84r4f_2084cf02-f2c8-4b63-a1f0-9934e9a8a2ae/kube-auth-proxy/0.log"
May 27 21:41:31.137067 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:31.137040    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df95f575-p6wj6_f5987bd4-82cb-432c-a36f-f37f00caaaf2/istio-proxy/0.log"
May 27 21:41:31.194472 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:31.194443    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58b6c8fdc7-p65kb_5a8e8c8d-a140-4d3e-991e-7b5000a3dd80/payload-processing/0.log"
May 27 21:41:31.716765 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:31.716739    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ds85j_162181ca-487b-47a0-b0e4-911370859701/serve-healthcheck-canary/0.log"
May 27 21:41:32.353225 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:32.353195    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s2fv5_02a18b4c-800c-461c-ab22-6ec305befca3/kube-rbac-proxy/0.log"
May 27 21:41:32.397774 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:32.397749    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s2fv5_02a18b4c-800c-461c-ab22-6ec305befca3/exporter/0.log"
May 27 21:41:32.419827 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:32.419798    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s2fv5_02a18b4c-800c-461c-ab22-6ec305befca3/extractor/0.log"
May 27 21:41:34.389091 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:34.389056    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-856948b99f-bjkjp_7037f41c-22ab-4a51-888d-40b2c8998c25/manager/0.log"
May 27 21:41:34.591521 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:34.591486    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-858dbf95b8-2jzbs_40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7/manager/1.log"
May 27 21:41:34.607478 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:34.607446    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-858dbf95b8-2jzbs_40e8e370-eb87-4e57-bcf1-ff8c0f13cfb7/manager/2.log"
May 27 21:41:34.671579 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:34.671488    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6678cd66dc-mk4qm_835def55-9d49-48a1-8c71-3755952cda77/manager/0.log"
May 27 21:41:34.707907 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:34.707883    2570 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t5vz8/perf-node-gather-daemonset-72m69"
May 27 21:41:34.721570 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:34.721536    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-68c4fbbd6f-7z5jw_053a60a0-4a39-4a72-9073-e76728d10fb8/postgres/0.log"
May 27 21:41:42.115424 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:42.115396    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4g469_dddcd173-827c-4681-ab31-637bc3186afb/kube-multus/0.log"
May 27 21:41:42.142991 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:42.142968    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7lpb9_766a48c1-61da-40cc-be8b-1688536b1722/kube-multus-additional-cni-plugins/0.log"
May 27 21:41:42.164832 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:42.164811    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7lpb9_766a48c1-61da-40cc-be8b-1688536b1722/egress-router-binary-copy/0.log"
May 27 21:41:42.186059 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:42.186035    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7lpb9_766a48c1-61da-40cc-be8b-1688536b1722/cni-plugins/0.log"
May 27 21:41:42.213013 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:42.212993    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7lpb9_766a48c1-61da-40cc-be8b-1688536b1722/bond-cni-plugin/0.log"
May 27 21:41:42.234349 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:42.234330    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7lpb9_766a48c1-61da-40cc-be8b-1688536b1722/routeoverride-cni/0.log"
May 27 21:41:42.259179 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:42.259159    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7lpb9_766a48c1-61da-40cc-be8b-1688536b1722/whereabouts-cni-bincopy/0.log"
May 27 21:41:42.281100 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:42.281079    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7lpb9_766a48c1-61da-40cc-be8b-1688536b1722/whereabouts-cni/0.log"
May 27 21:41:42.786591 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:42.786549    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nbt2c_375b9e24-ceeb-431a-9102-ca5ac59437cb/network-metrics-daemon/0.log"
May 27 21:41:42.808275 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:42.808249    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nbt2c_375b9e24-ceeb-431a-9102-ca5ac59437cb/kube-rbac-proxy/0.log"
May 27 21:41:43.965419 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:43.965384    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mwl2r_98aa4634-5b19-4d18-859e-457982a7dbb5/ovn-controller/0.log"
May 27 21:41:43.994313 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:43.994282    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mwl2r_98aa4634-5b19-4d18-859e-457982a7dbb5/ovn-acl-logging/0.log"
May 27 21:41:44.015123 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:44.015091    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mwl2r_98aa4634-5b19-4d18-859e-457982a7dbb5/kube-rbac-proxy-node/0.log"
May 27 21:41:44.035071 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:44.035046    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mwl2r_98aa4634-5b19-4d18-859e-457982a7dbb5/kube-rbac-proxy-ovn-metrics/0.log"
May 27 21:41:44.053958 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:44.053927    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mwl2r_98aa4634-5b19-4d18-859e-457982a7dbb5/northd/0.log"
May 27 21:41:44.079191 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:44.079170    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mwl2r_98aa4634-5b19-4d18-859e-457982a7dbb5/nbdb/0.log"
May 27 21:41:44.108390 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:44.108364    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mwl2r_98aa4634-5b19-4d18-859e-457982a7dbb5/sbdb/0.log"
May 27 21:41:44.216319 ip-10-0-128-93 kubenswrapper[2570]: I0527 21:41:44.216243    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mwl2r_98aa4634-5b19-4d18-859e-457982a7dbb5/ovnkube-controller/0.log"