May 28 00:11:20.246798 ip-10-0-128-163 systemd[1]: Starting Kubernetes Kubelet...
May 28 00:11:20.746897 ip-10-0-128-163 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 00:11:20.746897 ip-10-0-128-163 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 28 00:11:20.746897 ip-10-0-128-163 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 00:11:20.746897 ip-10-0-128-163 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 28 00:11:20.746897 ip-10-0-128-163 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 00:11:20.750354 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.750259    2576 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 28 00:11:20.758476 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758449    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 00:11:20.758476 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758470    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 00:11:20.758476 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758474    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 00:11:20.758476 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758477    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 00:11:20.758476 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758480    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 00:11:20.758476 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758484    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758487    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758490    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758493    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758495    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758498    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758501    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758519    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758523    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758526    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758531    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758534    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758536    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758539    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758542    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758544    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758551    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758554    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758557    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758559    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 00:11:20.758715 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758562    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758565    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758568    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758570    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758573    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758576    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758578    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758581    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758583    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758586    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758589    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758591    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758594    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758597    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758600    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758602    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758605    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758608    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758611    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 00:11:20.759188 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758613    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758617    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758622    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758625    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758628    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758630    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758633    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758635    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758638    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758641    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758644    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758647    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758649    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758652    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758654    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758657    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758660    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758662    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758664    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 00:11:20.759701 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758667    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758670    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758672    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758675    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758678    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758681    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758683    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758686    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758689    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758692    2576 feature_gate.go:328] unrecognized feature gate: Example2
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758695    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758698    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758700    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758703    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758707    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758712    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758715    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758718    2576 feature_gate.go:328] unrecognized feature gate: Example
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758722    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758725    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 00:11:20.760163 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758729    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758731    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.758734    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759164    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759169    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759172    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759175    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759178    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759180    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759183    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759186    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759188    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759191    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759194    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759197    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759200    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759203    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759205    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759208    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759211    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 00:11:20.760659 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759214    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759216    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759219    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759222    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759224    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759227    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759229    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759232    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759235    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759238    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759240    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759243    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759246    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759248    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759250    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759253    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759256    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759258    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759260    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 00:11:20.761129 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759265    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759269    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759272    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759274    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759277    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759280    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759282    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759285    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759287    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759290    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759293    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759295    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759299    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759302    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759304    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759307    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759310    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759312    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759314    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759317    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 00:11:20.761616 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759320    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759323    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759325    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759328    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759330    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759333    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759335    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759338    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759340    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759342    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759345    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759348    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759350    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759353    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759355    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759357    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759360    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759362    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759366    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759368    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 00:11:20.762134 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759371    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759374    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759376    2576 feature_gate.go:328] unrecognized feature gate: Example2
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759379    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759382    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759384    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759387    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759390    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759393    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.759396    2576 feature_gate.go:328] unrecognized feature gate: Example
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760271    2576 flags.go:64] FLAG: --address="0.0.0.0"
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760283    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760291    2576 flags.go:64] FLAG: --anonymous-auth="true"
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760296    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760301    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760305    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760309    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760314    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760318    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760321    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760325    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 28 00:11:20.762629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760329    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760332    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760335    2576 flags.go:64] FLAG: --cgroup-root=""
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760338    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760341    2576 flags.go:64] FLAG: --client-ca-file=""
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760344    2576 flags.go:64] FLAG: --cloud-config=""
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760346    2576 flags.go:64] FLAG: --cloud-provider="external"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.760350    2576 flags.go:64] FLAG: --cluster-dns="[]"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761084    2576 flags.go:64] FLAG: --cluster-domain=""
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761089    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761093    2576 flags.go:64] FLAG: --config-dir=""
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761096    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761100    2576 flags.go:64] FLAG: --container-log-max-files="5"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761104    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761107    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761111    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761114    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761117    2576 flags.go:64] FLAG: --contention-profiling="false"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761120    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761123    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761127    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761130    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761135    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761138    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761141    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 28 00:11:20.763140 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761144    2576 flags.go:64] FLAG: --enable-load-reader="false"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761150    2576 flags.go:64] FLAG: --enable-server="true"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761154    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761158    2576 flags.go:64] FLAG: --event-burst="100"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761161    2576 flags.go:64] FLAG: --event-qps="50"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761164    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761167    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761170    2576 flags.go:64] FLAG: --eviction-hard=""
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761174    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761177    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761180    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761183    2576 flags.go:64] FLAG: --eviction-soft=""
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761186    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761189    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761192    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761195    2576 flags.go:64] FLAG: --experimental-mounter-path=""
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761198    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761201    2576 flags.go:64] FLAG: --fail-swap-on="true"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761204    2576 flags.go:64] FLAG: --feature-gates=""
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761208    2576 flags.go:64] FLAG: --file-check-frequency="20s"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761211    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761214    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761218    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761222    2576 flags.go:64] FLAG: --healthz-port="10248"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761225    2576 flags.go:64] FLAG: --help="false"
May 28 00:11:20.763754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761228    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-128-163.ec2.internal"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761231    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761234    2576 flags.go:64] FLAG: --http-check-frequency="20s"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761237    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761241    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761244    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761247    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761250    2576 flags.go:64] FLAG: --image-service-endpoint=""
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761253    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761256    2576 flags.go:64] FLAG: --kube-api-burst="100"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761259    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761264    2576 flags.go:64] FLAG: --kube-api-qps="50"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761267    2576 flags.go:64] FLAG: --kube-reserved=""
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761270    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761273    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761276    2576 flags.go:64] FLAG: --kubelet-cgroups=""
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761280    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761283    2576 flags.go:64] FLAG: --lock-file=""
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761286    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761289    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761292    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761298    2576 flags.go:64] FLAG: --log-json-split-stream="false"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761301    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761304    2576 flags.go:64] FLAG: --log-text-split-stream="false"
May 28 00:11:20.764373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761307    2576 flags.go:64] FLAG: --logging-format="text"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761310    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761313    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761316    2576 flags.go:64] FLAG: --manifest-url=""
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761319    2576 flags.go:64] FLAG: --manifest-url-header=""
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761324    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761328    2576 flags.go:64] FLAG: --max-open-files="1000000"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761332    2576 flags.go:64] FLAG: --max-pods="110"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761335    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761338    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761341    2576 flags.go:64] FLAG: --memory-manager-policy="None"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761344    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761347    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761350    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761353    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761361    2576 flags.go:64] FLAG: --node-status-max-images="50"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761364    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761366    2576 flags.go:64] FLAG: --oom-score-adj="-999"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761370    2576 flags.go:64] FLAG: --pod-cidr=""
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761373    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761379    2576 flags.go:64] FLAG: --pod-manifest-path=""
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761382    2576 flags.go:64] FLAG: --pod-max-pids="-1"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761385    2576 flags.go:64] FLAG: --pods-per-core="0"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761388    2576 flags.go:64] FLAG: --port="10250"
May 28 00:11:20.764943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761391    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761394    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-079c82fe4332befa4"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761397    2576 flags.go:64] FLAG: --qos-reserved=""
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761400    2576 flags.go:64] FLAG: --read-only-port="10255"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761403    2576 flags.go:64] FLAG: --register-node="true"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761406    2576 flags.go:64] FLAG: --register-schedulable="true"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761409    2576 flags.go:64] FLAG: --register-with-taints=""
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761413    2576 flags.go:64] FLAG: --registry-burst="10"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761416    2576 flags.go:64] FLAG: --registry-qps="5"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761418    2576 flags.go:64] FLAG: --reserved-cpus=""
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761421    2576 flags.go:64] FLAG: --reserved-memory=""
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761425    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761428    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761431    2576 flags.go:64] FLAG: --rotate-certificates="false"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761434    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761437    2576 flags.go:64] FLAG: --runonce="false"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761439    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761442    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761445    2576 flags.go:64] FLAG: --seccomp-default="false"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761448    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761451    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761454    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761457    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761460    2576 flags.go:64] FLAG: --storage-driver-password="root"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761463    2576 flags.go:64] FLAG: --storage-driver-secure="false"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761465    2576 flags.go:64] FLAG: --storage-driver-table="stats"
May 28 00:11:20.765530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761468    2576 flags.go:64] FLAG: --storage-driver-user="root"
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761472    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761475    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761478    2576 flags.go:64] FLAG: --system-cgroups=""
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761480    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761486    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761488    2576 flags.go:64] FLAG: --tls-cert-file=""
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761491    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761496    2576 flags.go:64] FLAG: --tls-min-version=""
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761499    2576 flags.go:64] FLAG: --tls-private-key-file=""
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761518    2576 flags.go:64] FLAG: --topology-manager-policy="none"
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761523    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761526    2576 flags.go:64] FLAG: --topology-manager-scope="container"
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761529    2576 flags.go:64] FLAG: --v="2"
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761534    2576 flags.go:64] FLAG: --version="false"
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761538    2576 flags.go:64] FLAG: --vmodule=""
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761542    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.761546    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761647    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761651    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761654    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761657    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761660    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 00:11:20.766156 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761663    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761666    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761669    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761672    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761675    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761678    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761681    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761684    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761687    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761690    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761693    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761696    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761699    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761701    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761704    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761707    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761710    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761712    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761715    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 00:11:20.766729 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761719    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761722    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761725    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761727    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761730    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761732    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761735    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761738    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761741    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761743    2576 feature_gate.go:328] unrecognized feature gate: Example2
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761746    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761748    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761751    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761755    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761758    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761761    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761764    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761767    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761770    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 00:11:20.767243 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761772    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761775    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761778    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761781    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761783    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761786    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761789    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761791    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761794    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761796    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761799    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761801    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761804    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761808    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761811    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761814    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761816    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761819    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761822    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761825    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 00:11:20.767774 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761828    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761830    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761833    2576 feature_gate.go:328] unrecognized feature gate: Example
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761836    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761838    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761841    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761844    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761846    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761849    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761851    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761854    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761856    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761859    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761862    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761864    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761867    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761870    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761873    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761876    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761879    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 00:11:20.768274 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761882    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 00:11:20.768780 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761884    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 00:11:20.768780 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.761887    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 00:11:20.768780 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.762741    2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 00:11:20.769754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.769735    2576 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 28 00:11:20.769786 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.769756    2576 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 28 00:11:20.769823 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769814    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 00:11:20.769823 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769823    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769826    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769830    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769833    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769836    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769838    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769841    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769843    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769846    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769849    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769852    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769854    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769857    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769859    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769862    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769865    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769867    2576 feature_gate.go:328] unrecognized feature gate: Example2
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769870    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769872    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769875    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 00:11:20.769877 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769877    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769880    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769887    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769890    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769895    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769899    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769901    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769904    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769906    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769909    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769911    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769914    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769916    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769919    2576 feature_gate.go:328] unrecognized feature gate: Example
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769922    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769924    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769927    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769929    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769932    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 00:11:20.770361 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769934    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769937    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769939    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769942    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769945    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769947    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769950    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769953    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769955    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769958    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769960    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769964    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769966    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769969    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769971    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769975    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769977    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769980    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769983    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769985    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 00:11:20.770852 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769988    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769990    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769993    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.769996    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770000    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770004    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770007    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770011    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770014    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770016    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770020    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770022    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770025    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770028    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770031    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770034    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770037    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770040    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770043    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 00:11:20.771341 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770045    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 00:11:20.771822 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770048    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 00:11:20.771822 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770051    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 00:11:20.771822 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770054    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 00:11:20.771822 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770057    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 00:11:20.771822 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770059    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 00:11:20.771822 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770062    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 00:11:20.771822 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.770067    2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 00:11:20.771822 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770180    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 00:11:20.771822 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770185    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 00:11:20.771822 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770187    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 00:11:20.771822 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770191    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 00:11:20.771822 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770194    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 00:11:20.771822 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770197    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 00:11:20.771822 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770200    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 00:11:20.771822 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770202    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770205    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770207    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770210    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770213    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770215    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770217    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770220    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770223    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770225    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770228    2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770230    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770233    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770235    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770238    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770241    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770243    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770246    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770248    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770251    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 00:11:20.772200 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770253    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770256    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770258    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770261    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770263    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770266    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770269    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770271    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770274    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770276    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770279    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770282    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770285    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770287    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770289    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770292    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770294    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770297    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770299    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770302    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 00:11:20.772691 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770304    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770307    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770309    2576 feature_gate.go:328] unrecognized feature gate: Example2
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770312    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770314    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770317    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770319    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770322    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770324    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770327    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770329    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770333    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770337    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770340    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770343    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770346    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770348    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770351    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770354    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 00:11:20.773171 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770356    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770359    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770361    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770364    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770372    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770376    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770379    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770382    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770384    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770387    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770389    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770392    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770394    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770397    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770399    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770402    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770404    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770407    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770409    2576 feature_gate.go:328] unrecognized feature gate: Example
May 28 00:11:20.773643 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:20.770412    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 00:11:20.774089 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.770416    2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 00:11:20.774089 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.771325    2576 server.go:962] "Client rotation is on, will bootstrap in background"
May 28 00:11:20.774089 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.773791    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 28 00:11:20.774964 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.774951    2576 server.go:1019] "Starting client certificate rotation"
May 28 00:11:20.775072 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.775054    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 28 00:11:20.775102 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.775091    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 28 00:11:20.803700 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.803677    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 28 00:11:20.804914 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.804893    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 28 00:11:20.823422 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.823376    2576 log.go:25] "Validated CRI v1 runtime API"
May 28 00:11:20.830471 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.830455    2576 log.go:25] "Validated CRI v1 image API"
May 28 00:11:20.833403 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.833382    2576 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 28 00:11:20.837661 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.837640    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8ecdf6c9-2e9c-4e56-b131-c5cee9f87b54:/dev/nvme0n1p4 e7fc0826-9967-45f9-a651-c0489346b98a:/dev/nvme0n1p3]
May 28 00:11:20.837737 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.837661    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 28 00:11:20.840695 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.840554    2576 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 28 00:11:20.841918 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.841807    2576 manager.go:217] Machine: {Timestamp:2026-05-28 00:11:20.841284344 +0000 UTC m=+0.462316351 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098644 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec270954b89aef08b79112498f0fea74 SystemUUID:ec270954-b89a-ef08-b791-12498f0fea74 BootID:0295f205-22fe-4f4a-a9c7-801fc063b1b7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:65:61:73:c1:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:65:61:73:c1:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:bf:79:fa:21:12 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 28 00:11:20.842568 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.842557    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 28 00:11:20.842689 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.842678    2576 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 28 00:11:20.845696 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.845670    2576 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 28 00:11:20.845863 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.845698    2576 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-163.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 28 00:11:20.845945 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.845873    2576 topology_manager.go:138] "Creating topology manager with none policy"
May 28 00:11:20.845945 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.845887    2576 container_manager_linux.go:306] "Creating device plugin manager"
May 28 00:11:20.845945 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.845904    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 28 00:11:20.847019 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.847007    2576 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 28 00:11:20.848955 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.848942    2576 state_mem.go:36] "Initialized new in-memory state store"
May 28 00:11:20.849092 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.849080    2576 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 28 00:11:20.851830 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.851818    2576 kubelet.go:491] "Attempting to sync node with API server"
May 28 00:11:20.851888 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.851836    2576 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 28 00:11:20.851888 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.851853    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 28 00:11:20.851888 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.851867    2576 kubelet.go:397] "Adding apiserver pod source"
May 28 00:11:20.851888 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.851880    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 28 00:11:20.853132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.853118    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 28 00:11:20.853207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.853140    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 28 00:11:20.856596 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.856578    2576 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 28 00:11:20.859013 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.858999    2576 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 28 00:11:20.861362 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.861350    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 28 00:11:20.861406 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.861367    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 28 00:11:20.861406 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.861374    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 28 00:11:20.861406 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.861380    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 28 00:11:20.861406 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.861389    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 28 00:11:20.861406 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.861394    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 28 00:11:20.861406 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.861400    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 28 00:11:20.861406 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.861406    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 28 00:11:20.861608 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.861412    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 28 00:11:20.861608 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.861418    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 28 00:11:20.861608 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.861427    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 28 00:11:20.861608 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.861439    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 28 00:11:20.863578 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.863567    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 28 00:11:20.863614 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.863579    2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 28 00:11:20.865715 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:20.865689    2576 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 28 00:11:20.865715 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:20.865692    2576 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 28 00:11:20.866890 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.866874    2576 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 28 00:11:20.866957 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.866876    2576 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-163.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 28 00:11:20.866957 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.866907    2576 server.go:1295] "Started kubelet"
May 28 00:11:20.867051 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.866988    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 28 00:11:20.867113 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.867013    2576 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 28 00:11:20.867146 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.867137    2576 server_v1.go:47] "podresources" method="list" useActivePods=true
May 28 00:11:20.867742 ip-10-0-128-163 systemd[1]: Started Kubernetes Kubelet.
May 28 00:11:20.868648 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.868635    2576 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 28 00:11:20.869859 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.869845    2576 server.go:317] "Adding debug handlers to kubelet server"
May 28 00:11:20.876902 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.876886    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 28 00:11:20.878953 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.878918    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 28 00:11:20.879142 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.879125    2576 volume_manager.go:295] "The desired_state_of_world populator starts"
May 28 00:11:20.879213 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.879145    2576 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 28 00:11:20.879679 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.879663    2576 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 28 00:11:20.879784 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.879752    2576 reconstruct.go:97] "Volume reconstruction finished"
May 28 00:11:20.879784 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.879764    2576 reconciler.go:26] "Reconciler: start to sync state"
May 28 00:11:20.880032 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:20.880001    2576 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 28 00:11:20.880582 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:20.880562    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:20.880697 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:20.879492    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-163.ec2.internal.18b3923fe52b352e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-163.ec2.internal,UID:ip-10-0-128-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-163.ec2.internal,},FirstTimestamp:2026-05-28 00:11:20.866886958 +0000 UTC m=+0.487918962,LastTimestamp:2026-05-28 00:11:20.866886958 +0000 UTC m=+0.487918962,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-163.ec2.internal,}"
May 28 00:11:20.880972 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.880953    2576 factory.go:153] Registering CRI-O factory
May 28 00:11:20.881047 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.880981    2576 factory.go:223] Registration of the crio container factory successfully
May 28 00:11:20.881047 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.881033    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 28 00:11:20.881047 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.881042    2576 factory.go:55] Registering systemd factory
May 28 00:11:20.881185 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.881051    2576 factory.go:223] Registration of the systemd container factory successfully
May 28 00:11:20.881185 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.881068    2576 factory.go:103] Registering Raw factory
May 28 00:11:20.881185 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.881095    2576 manager.go:1196] Started watching for new ooms in manager
May 28 00:11:20.882733 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.882719    2576 manager.go:319] Starting recovery of all containers
May 28 00:11:20.883168 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:20.883139    2576 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 28 00:11:20.883275 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:20.883237    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 28 00:11:20.892699 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.892540    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l8mlq"
May 28 00:11:20.893193 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.893179    2576 manager.go:324] Recovery completed
May 28 00:11:20.897013 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.897001    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 00:11:20.898711 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.898696    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientMemory"
May 28 00:11:20.898799 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.898728    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasNoDiskPressure"
May 28 00:11:20.898799 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.898744    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientPID"
May 28 00:11:20.899179 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.899162    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l8mlq"
May 28 00:11:20.899243 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.899229    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 28 00:11:20.899277 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.899244    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 28 00:11:20.899277 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.899264    2576 state_mem.go:36] "Initialized new in-memory state store"
May 28 00:11:20.900307 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:20.900243    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-163.ec2.internal.18b3923fe710d2b9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-163.ec2.internal,UID:ip-10-0-128-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-163.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-163.ec2.internal,},FirstTimestamp:2026-05-28 00:11:20.898712249 +0000 UTC m=+0.519744254,LastTimestamp:2026-05-28 00:11:20.898712249 +0000 UTC m=+0.519744254,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-163.ec2.internal,}"
May 28 00:11:20.904405 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.904391    2576 policy_none.go:49] "None policy: Start"
May 28 00:11:20.904471 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.904409    2576 memory_manager.go:186] "Starting memorymanager" policy="None"
May 28 00:11:20.904471 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.904420    2576 state_mem.go:35] "Initializing new in-memory state store"
May 28 00:11:20.943102 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.943088    2576 manager.go:341] "Starting Device Plugin manager"
May 28 00:11:20.965277 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:20.943188    2576 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 28 00:11:20.965277 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.943203    2576 server.go:85] "Starting device plugin registration server"
May 28 00:11:20.965277 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.943450    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
May 28 00:11:20.965277 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.943461    2576 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 28 00:11:20.965277 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.943558    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 28 00:11:20.965277 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.943630    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 28 00:11:20.965277 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:20.943638    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 28 00:11:20.965277 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:20.944196    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 28 00:11:20.965277 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:20.944229    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:21.010033 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.009956    2576 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 28 00:11:21.011227 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.011210    2576 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 28 00:11:21.011227 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.011230    2576 status_manager.go:230] "Starting to sync pod status with apiserver"
May 28 00:11:21.011355 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.011248    2576 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 28 00:11:21.011355 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.011255    2576 kubelet.go:2451] "Starting kubelet main sync loop"
May 28 00:11:21.011355 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:21.011297    2576 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 28 00:11:21.018174 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.018157    2576 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 00:11:21.044290 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.044268    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 00:11:21.045195 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.045179    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientMemory"
May 28 00:11:21.045278 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.045206    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasNoDiskPressure"
May 28 00:11:21.045278 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.045217    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientPID"
May 28 00:11:21.045278 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.045239    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-163.ec2.internal"
May 28 00:11:21.057765 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.057748    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-163.ec2.internal"
May 28 00:11:21.057813 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:21.057770    2576 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-163.ec2.internal\": node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:21.074292 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:21.074273    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:21.112316 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.112273    2576 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal"]
May 28 00:11:21.112424 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.112356    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 00:11:21.113928 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.113913    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientMemory"
May 28 00:11:21.114002 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.113941    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasNoDiskPressure"
May 28 00:11:21.114002 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.113952    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientPID"
May 28 00:11:21.115014 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.115002    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 00:11:21.115151 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.115137    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
May 28 00:11:21.115203 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.115176    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 00:11:21.115735 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.115719    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientMemory"
May 28 00:11:21.115735 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.115726    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientMemory"
May 28 00:11:21.115890 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.115744    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasNoDiskPressure"
May 28 00:11:21.115890 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.115751    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasNoDiskPressure"
May 28 00:11:21.115890 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.115756    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientPID"
May 28 00:11:21.115890 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.115762    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientPID"
May 28 00:11:21.116747 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.116729    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal"
May 28 00:11:21.116835 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.116761    2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 00:11:21.117404 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.117387    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientMemory"
May 28 00:11:21.117493 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.117414    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasNoDiskPressure"
May 28 00:11:21.117493 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.117424    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeHasSufficientPID"
May 28 00:11:21.144165 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:21.144143    2576 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-163.ec2.internal\" not found" node="ip-10-0-128-163.ec2.internal"
May 28 00:11:21.148640 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:21.148622    2576 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-163.ec2.internal\" not found" node="ip-10-0-128-163.ec2.internal"
May 28 00:11:21.175044 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:21.175023    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:21.181934 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.181915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c3a83805b8518efc6135ad03a3963fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal\" (UID: \"2c3a83805b8518efc6135ad03a3963fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
May 28 00:11:21.181988 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.181942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/276f8146c60adf430fac9496a6d82374-config\") pod \"kube-apiserver-proxy-ip-10-0-128-163.ec2.internal\" (UID: \"276f8146c60adf430fac9496a6d82374\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal"
May 28 00:11:21.181988 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.181959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c3a83805b8518efc6135ad03a3963fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal\" (UID: \"2c3a83805b8518efc6135ad03a3963fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
May 28 00:11:21.275543 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:21.275454    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:21.282932 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.282908    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c3a83805b8518efc6135ad03a3963fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal\" (UID: \"2c3a83805b8518efc6135ad03a3963fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
May 28 00:11:21.282995 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.282940    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c3a83805b8518efc6135ad03a3963fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal\" (UID: \"2c3a83805b8518efc6135ad03a3963fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
May 28 00:11:21.282995 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.282961    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/276f8146c60adf430fac9496a6d82374-config\") pod \"kube-apiserver-proxy-ip-10-0-128-163.ec2.internal\" (UID: \"276f8146c60adf430fac9496a6d82374\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal"
May 28 00:11:21.283070 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.283008    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c3a83805b8518efc6135ad03a3963fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal\" (UID: \"2c3a83805b8518efc6135ad03a3963fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
May 28 00:11:21.283070 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.283012    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c3a83805b8518efc6135ad03a3963fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal\" (UID: \"2c3a83805b8518efc6135ad03a3963fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
May 28 00:11:21.283070 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.283051    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/276f8146c60adf430fac9496a6d82374-config\") pod \"kube-apiserver-proxy-ip-10-0-128-163.ec2.internal\" (UID: \"276f8146c60adf430fac9496a6d82374\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal"
May 28 00:11:21.376377 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:21.376334    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:21.446916 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.446874    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
May 28 00:11:21.451499 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.451481    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal"
May 28 00:11:21.476494 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:21.476463    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:21.577392 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:21.577305    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:21.677909 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:21.677873    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:21.774519 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.774473    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 28 00:11:21.775152 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.774649    2576 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 28 00:11:21.778637 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:21.778619    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:21.877378 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.877353    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 28 00:11:21.878812 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:21.878788    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:21.880499 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.880482    2576 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 00:11:21.898717 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.898693    2576 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 28 00:11:21.901870 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.901837    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-27 00:06:20 +0000 UTC" deadline="2027-12-18 20:57:48.537284717 +0000 UTC"
May 28 00:11:21.901870 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.901860    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13676h46m26.635427049s"
May 28 00:11:21.978401 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:21.978369    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3a83805b8518efc6135ad03a3963fc.slice/crio-864e5709d2283f5048d3d26fe93a6e43153aef022c77fc9898dde9a4e494d544 WatchSource:0}: Error finding container 864e5709d2283f5048d3d26fe93a6e43153aef022c77fc9898dde9a4e494d544: Status 404 returned error can't find the container with id 864e5709d2283f5048d3d26fe93a6e43153aef022c77fc9898dde9a4e494d544
May 28 00:11:21.978734 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:21.978711    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276f8146c60adf430fac9496a6d82374.slice/crio-a0111e36cda9190f0a1975a21e0826bbc1d40c40261fbba2316ce2706d08f4ba WatchSource:0}: Error finding container a0111e36cda9190f0a1975a21e0826bbc1d40c40261fbba2316ce2706d08f4ba: Status 404 returned error can't find the container with id a0111e36cda9190f0a1975a21e0826bbc1d40c40261fbba2316ce2706d08f4ba
May 28 00:11:21.979837 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:21.979754    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:21.981970 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:21.981955    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 00:11:22.002939 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.002918    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kbc9c"
May 28 00:11:22.014320 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.014267    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal" event={"ID":"276f8146c60adf430fac9496a6d82374","Type":"ContainerStarted","Data":"a0111e36cda9190f0a1975a21e0826bbc1d40c40261fbba2316ce2706d08f4ba"}
May 28 00:11:22.015320 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.015287    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal" event={"ID":"2c3a83805b8518efc6135ad03a3963fc","Type":"ContainerStarted","Data":"864e5709d2283f5048d3d26fe93a6e43153aef022c77fc9898dde9a4e494d544"}
May 28 00:11:22.030008 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.029989    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kbc9c"
May 28 00:11:22.080333 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:22.080310    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:22.085987 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.085968    2576 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 00:11:22.180737 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:22.180707    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:22.281137 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:22.281102    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:22.381912 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:22.381883    2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-163.ec2.internal\" not found"
May 28 00:11:22.388636 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.388615    2576 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 00:11:22.479974 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.479891    2576 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal"
May 28 00:11:22.500202 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.500176    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 28 00:11:22.501348 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.501334    2576 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal"
May 28 00:11:22.515219 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.515200    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 28 00:11:22.852942 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.852910    2576 apiserver.go:52] "Watching apiserver"
May 28 00:11:22.862071 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.862044    2576 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 28 00:11:22.862763 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.862731    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl","openshift-cluster-node-tuning-operator/tuned-zwmf5","openshift-dns/node-resolver-tmc88","openshift-image-registry/node-ca-mvfbv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal","openshift-multus/network-metrics-daemon-6b84s","openshift-network-diagnostics/network-check-target-xq6dj","kube-system/konnectivity-agent-jxqqz","openshift-multus/multus-62mcg","openshift-multus/multus-additional-cni-plugins-r9lmd","openshift-network-operator/iptables-alerter-bvcpc","openshift-ovn-kubernetes/ovnkube-node-8gct7"]
May 28 00:11:22.865338 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.865313    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:22.865442 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:22.865408    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:22.867604 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.867468    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.869672 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.869612    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.870578 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.870522    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 28 00:11:22.871755 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.871734    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tmc88"
May 28 00:11:22.873928 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.873908    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mvfbv"
May 28 00:11:22.874436 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.874412    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 28 00:11:22.874541 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.874455    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 28 00:11:22.874541 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.874474    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7nqc7\""
May 28 00:11:22.876321 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.876112    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 28 00:11:22.876607 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.876588    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 28 00:11:22.876839 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.876815    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 28 00:11:22.877151 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.877136    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zswfr\""
May 28 00:11:22.877611 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.877593    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 28 00:11:22.882124 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.882093    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ktw8b\""
May 28 00:11:22.882229 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.882213    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 28 00:11:22.882461 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.882439    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9bg6w\""
May 28 00:11:22.883161 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.882572    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 28 00:11:22.883161 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.882976    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 28 00:11:22.883161 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.883103    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-62mcg"
May 28 00:11:22.883364 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.883341    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:22.883457 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:22.883420    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:22.883542 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.883471    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jxqqz"
May 28 00:11:22.887368 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.887349    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:22.888147 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.888128    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jl9qk\""
May 28 00:11:22.888236 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.888152    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 28 00:11:22.889699 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.889682    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bvcpc"
May 28 00:11:22.890316 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.890298    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vmq8c\""
May 28 00:11:22.890410 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.890375    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 28 00:11:22.890705 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.890685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-socket-dir\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.890777 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.890708    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 28 00:11:22.890777 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.890723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vn5z\" (UniqueName: \"kubernetes.io/projected/637d10d6-892a-46db-ac00-41e0294451d3-kube-api-access-6vn5z\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.890777 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.890749    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-var-lib-cni-multus\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.890777 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.890772    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjbpc\" (UniqueName: \"kubernetes.io/projected/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-kube-api-access-xjbpc\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.890963 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.890829    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d74ab08c-2275-4054-9365-ba62ea412c17-hosts-file\") pod \"node-resolver-tmc88\" (UID: \"d74ab08c-2275-4054-9365-ba62ea412c17\") " pod="openshift-dns/node-resolver-tmc88"
May 28 00:11:22.890963 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.890851    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 28 00:11:22.891061 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.890856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-run-multus-certs\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.891061 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891038    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-modprobe-d\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.891165 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891066    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-systemd\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.891165 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-tmp\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.891165 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86gdr\" (UniqueName: \"kubernetes.io/projected/d74ab08c-2275-4054-9365-ba62ea412c17-kube-api-access-86gdr\") pod \"node-resolver-tmc88\" (UID: \"d74ab08c-2275-4054-9365-ba62ea412c17\") " pod="openshift-dns/node-resolver-tmc88"
May 28 00:11:22.891165 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmbbg\" (UniqueName: \"kubernetes.io/projected/653f150e-1c4b-4dc5-a088-6ab0ad385350-kube-api-access-vmbbg\") pod \"network-metrics-daemon-6b84s\" (UID: \"653f150e-1c4b-4dc5-a088-6ab0ad385350\") " pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:22.891165 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2f133ad-ace2-417a-a450-7f917265d112-host\") pod \"node-ca-mvfbv\" (UID: \"a2f133ad-ace2-417a-a450-7f917265d112\") " pod="openshift-image-registry/node-ca-mvfbv"
May 28 00:11:22.891403 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891179    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqzwc\" (UniqueName: \"kubernetes.io/projected/a2f133ad-ace2-417a-a450-7f917265d112-kube-api-access-wqzwc\") pod \"node-ca-mvfbv\" (UID: \"a2f133ad-ace2-417a-a450-7f917265d112\") " pod="openshift-image-registry/node-ca-mvfbv"
May 28 00:11:22.891403 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891213    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-hostroot\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.891403 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891237    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-sysconfig\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.891403 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891312    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-kubernetes\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.891403 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2435a2d-273d-4891-918d-377147d3eb63-cni-binary-copy\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.891403 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891369    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-multus-socket-dir-parent\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.891702 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891408    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-874zc\" (UniqueName: \"kubernetes.io/projected/a2435a2d-273d-4891-918d-377147d3eb63-kube-api-access-874zc\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.891702 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891437    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-run\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.891702 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d74ab08c-2275-4054-9365-ba62ea412c17-tmp-dir\") pod \"node-resolver-tmc88\" (UID: \"d74ab08c-2275-4054-9365-ba62ea412c17\") " pod="openshift-dns/node-resolver-tmc88"
May 28 00:11:22.891702 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891488    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ec0d0ea-61c0-4df1-9eb0-a78662baca1d-agent-certs\") pod \"konnectivity-agent-jxqqz\" (UID: \"4ec0d0ea-61c0-4df1-9eb0-a78662baca1d\") " pod="kube-system/konnectivity-agent-jxqqz"
May 28 00:11:22.891702 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891526    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ec0d0ea-61c0-4df1-9eb0-a78662baca1d-konnectivity-ca\") pod \"konnectivity-agent-jxqqz\" (UID: \"4ec0d0ea-61c0-4df1-9eb0-a78662baca1d\") " pod="kube-system/konnectivity-agent-jxqqz"
May 28 00:11:22.891702 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891552    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.891702 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-device-dir\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.891702 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891599    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-run-k8s-cni-cncf-io\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.891702 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891632    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-run-netns\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.891702 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-var-lib-kubelet\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.891702 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt654\" (UniqueName: \"kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654\") pod \"network-check-target-xq6dj\" (UID: \"2e67c0ae-9c40-46d4-91a8-a5577a46bd2f\") " pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891711    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-registration-dir\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891733    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-sys-fs\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-multus-cni-dir\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-var-lib-cni-bin\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891802    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-multus-conf-dir\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891826    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-sysctl-conf\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891850    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-sys\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-lib-modules\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891914    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs\") pod \"network-metrics-daemon-6b84s\" (UID: \"653f150e-1c4b-4dc5-a088-6ab0ad385350\") " pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891937    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2f133ad-ace2-417a-a450-7f917265d112-serviceca\") pod \"node-ca-mvfbv\" (UID: \"a2f133ad-ace2-417a-a450-7f917265d112\") " pod="openshift-image-registry/node-ca-mvfbv"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891963    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-cnibin\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.891983    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-os-release\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.892012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-var-lib-kubelet\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.892045    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2435a2d-273d-4891-918d-377147d3eb63-multus-daemon-config\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.892079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-sysctl-d\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.892124    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-tuned\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.892207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.892159    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.892976 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.892197    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.892976 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.892209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-system-cni-dir\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.892976 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.892242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-etc-kubernetes\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.892976 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.892264    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-host\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.893329 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.893313    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 28 00:11:22.893421 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.893403    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 28 00:11:22.893475 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.893426    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 28 00:11:22.893475 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.893453    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 28 00:11:22.893591 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.893365    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j9djt\""
May 28 00:11:22.893591 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.893355    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 28 00:11:22.893696 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.893384    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xm8tm\""
May 28 00:11:22.893696 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.893313    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 28 00:11:22.895539 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.895520    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 28 00:11:22.898285 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.898266    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 28 00:11:22.899062 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.899043    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 28 00:11:22.899336 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.899312    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 28 00:11:22.899545 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.899529    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 28 00:11:22.899765 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.899749    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 28 00:11:22.899870 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.899855    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 28 00:11:22.901839 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.901822    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bj4w5\""
May 28 00:11:22.980941 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.980914    2576 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 28 00:11:22.992598 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992566    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-run-netns\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.992732 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992610    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5d293d3-caa4-44cf-bae5-219a9e493213-os-release\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:22.992732 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992638    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xjbpc\" (UniqueName: \"kubernetes.io/projected/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-kube-api-access-xjbpc\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.992732 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992662    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d74ab08c-2275-4054-9365-ba62ea412c17-hosts-file\") pod \"node-resolver-tmc88\" (UID: \"d74ab08c-2275-4054-9365-ba62ea412c17\") " pod="openshift-dns/node-resolver-tmc88"
May 28 00:11:22.992732 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992688    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-run-netns\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.992732 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992688    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d293d3-caa4-44cf-bae5-219a9e493213-system-cni-dir\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:22.992984 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992738    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-registration-dir\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.992984 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992764    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-var-lib-cni-bin\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.992984 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992777    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d74ab08c-2275-4054-9365-ba62ea412c17-hosts-file\") pod \"node-resolver-tmc88\" (UID: \"d74ab08c-2275-4054-9365-ba62ea412c17\") " pod="openshift-dns/node-resolver-tmc88"
May 28 00:11:22.992984 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992792    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-lib-modules\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.992984 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992816    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-tuned\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.992984 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992844    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-var-lib-openvswitch\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.992984 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992874    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs\") pod \"network-metrics-daemon-6b84s\" (UID: \"653f150e-1c4b-4dc5-a088-6ab0ad385350\") " pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:22.992984 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992885    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-var-lib-cni-bin\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.992984 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992898    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2f133ad-ace2-417a-a450-7f917265d112-serviceca\") pod \"node-ca-mvfbv\" (UID: \"a2f133ad-ace2-417a-a450-7f917265d112\") " pod="openshift-image-registry/node-ca-mvfbv"
May 28 00:11:22.992984 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992922    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-os-release\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.992984 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992947    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-var-lib-kubelet\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.992984 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992947    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-registration-dir\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.992984 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.992977    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-sysctl-d\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993010    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw887\" (UniqueName: \"kubernetes.io/projected/bd2bcc29-a5ec-4d13-832d-f74b151a4c3d-kube-api-access-cw887\") pod \"iptables-alerter-bvcpc\" (UID: \"bd2bcc29-a5ec-4d13-832d-f74b151a4c3d\") " pod="openshift-network-operator/iptables-alerter-bvcpc"
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993033    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-run-openvswitch\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:22.993059    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993081    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993135    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:22.993144    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs podName:653f150e-1c4b-4dc5-a088-6ab0ad385350 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:23.493098009 +0000 UTC m=+3.114130003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs") pod "network-metrics-daemon-6b84s" (UID: "653f150e-1c4b-4dc5-a088-6ab0ad385350") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993162    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-system-cni-dir\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993190    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/39bb9abb-9e4b-4adb-a92c-cb7c18563333-ovnkube-config\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/39bb9abb-9e4b-4adb-a92c-cb7c18563333-ovn-node-metrics-cert\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993244    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6vn5z\" (UniqueName: \"kubernetes.io/projected/637d10d6-892a-46db-ac00-41e0294451d3-kube-api-access-6vn5z\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993252    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-sysctl-d\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993273    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-var-lib-cni-multus\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993301    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-slash\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993318    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-system-cni-dir\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993318    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 28 00:11:22.993794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993327    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-log-socket\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993354    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcjzk\" (UniqueName: \"kubernetes.io/projected/39bb9abb-9e4b-4adb-a92c-cb7c18563333-kube-api-access-qcjzk\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993379    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-systemd\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993404    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-86gdr\" (UniqueName: \"kubernetes.io/projected/d74ab08c-2275-4054-9365-ba62ea412c17-kube-api-access-86gdr\") pod \"node-resolver-tmc88\" (UID: \"d74ab08c-2275-4054-9365-ba62ea412c17\") " pod="openshift-dns/node-resolver-tmc88"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993434    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5d293d3-caa4-44cf-bae5-219a9e493213-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-run-systemd\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993489    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2f133ad-ace2-417a-a450-7f917265d112-host\") pod \"node-ca-mvfbv\" (UID: \"a2f133ad-ace2-417a-a450-7f917265d112\") " pod="openshift-image-registry/node-ca-mvfbv"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993529    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-kubernetes\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993554    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d74ab08c-2275-4054-9365-ba62ea412c17-tmp-dir\") pod \"node-resolver-tmc88\" (UID: \"d74ab08c-2275-4054-9365-ba62ea412c17\") " pod="openshift-dns/node-resolver-tmc88"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993581    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd2bcc29-a5ec-4d13-832d-f74b151a4c3d-host-slash\") pod \"iptables-alerter-bvcpc\" (UID: \"bd2bcc29-a5ec-4d13-832d-f74b151a4c3d\") " pod="openshift-network-operator/iptables-alerter-bvcpc"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993606    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-multus-socket-dir-parent\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993631    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-run\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993636    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-var-lib-kubelet\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993655    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/39bb9abb-9e4b-4adb-a92c-cb7c18563333-env-overrides\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993683    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993708    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-device-dir\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993712    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-os-release\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.994612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993733    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-run-k8s-cni-cncf-io\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993743    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2f133ad-ace2-417a-a450-7f917265d112-serviceca\") pod \"node-ca-mvfbv\" (UID: \"a2f133ad-ace2-417a-a450-7f917265d112\") " pod="openshift-image-registry/node-ca-mvfbv"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993760    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-var-lib-kubelet\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993786    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kt654\" (UniqueName: \"kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654\") pod \"network-check-target-xq6dj\" (UID: \"2e67c0ae-9c40-46d4-91a8-a5577a46bd2f\") " pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-systemd-units\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993857    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-run-netns\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.993884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-node-log\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.994692    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-sys-fs\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.994743    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-multus-cni-dir\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.994761    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-systemd\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.994786    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-multus-conf-dir\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.994814    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-sys-fs\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.994826    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-sysctl-conf\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.994942    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-sysctl-conf\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.994985    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-sys\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995022    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-var-lib-cni-multus\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995033    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-sys\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995103    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-multus-cni-dir\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.995379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995113    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d74ab08c-2275-4054-9365-ba62ea412c17-tmp-dir\") pod \"node-resolver-tmc88\" (UID: \"d74ab08c-2275-4054-9365-ba62ea412c17\") " pod="openshift-dns/node-resolver-tmc88"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.994744    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-kubernetes\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995181    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-multus-socket-dir-parent\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995202    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-lib-modules\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995236    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-var-lib-kubelet\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995406    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-device-dir\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995249    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-multus-conf-dir\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995294    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-run\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995446    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995490    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-run-k8s-cni-cncf-io\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995545    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5d293d3-caa4-44cf-bae5-219a9e493213-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995572    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2f133ad-ace2-417a-a450-7f917265d112-host\") pod \"node-ca-mvfbv\" (UID: \"a2f133ad-ace2-417a-a450-7f917265d112\") " pod="openshift-image-registry/node-ca-mvfbv"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5d293d3-caa4-44cf-bae5-219a9e493213-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995631    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-cnibin\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995671    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2435a2d-273d-4891-918d-377147d3eb63-multus-daemon-config\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-run-ovn-kubernetes\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995739    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-etc-kubernetes\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995743    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-cnibin\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.996132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995765    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-host\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995815    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-etc-openvswitch\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995825    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-host\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995849    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-socket-dir\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995880    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-etc-kubernetes\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd2bcc29-a5ec-4d13-832d-f74b151a4c3d-iptables-alerter-script\") pod \"iptables-alerter-bvcpc\" (UID: \"bd2bcc29-a5ec-4d13-832d-f74b151a4c3d\") " pod="openshift-network-operator/iptables-alerter-bvcpc"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995922    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrzf\" (UniqueName: \"kubernetes.io/projected/d5d293d3-caa4-44cf-bae5-219a9e493213-kube-api-access-tjrzf\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995957    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-run-multus-certs\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.995988    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-modprobe-d\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996011    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-tmp\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996050    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-run-ovn\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996083    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vmbbg\" (UniqueName: \"kubernetes.io/projected/653f150e-1c4b-4dc5-a088-6ab0ad385350-kube-api-access-vmbbg\") pod \"network-metrics-daemon-6b84s\" (UID: \"653f150e-1c4b-4dc5-a088-6ab0ad385350\") " pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996114    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wqzwc\" (UniqueName: \"kubernetes.io/projected/a2f133ad-ace2-417a-a450-7f917265d112-kube-api-access-wqzwc\") pod \"node-ca-mvfbv\" (UID: \"a2f133ad-ace2-417a-a450-7f917265d112\") " pod="openshift-image-registry/node-ca-mvfbv"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996144    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-hostroot\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996171    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-sysconfig\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996200    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ec0d0ea-61c0-4df1-9eb0-a78662baca1d-agent-certs\") pod \"konnectivity-agent-jxqqz\" (UID: \"4ec0d0ea-61c0-4df1-9eb0-a78662baca1d\") " pod="kube-system/konnectivity-agent-jxqqz"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5d293d3-caa4-44cf-bae5-219a9e493213-cnibin\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:22.996927 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996312    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-cni-bin\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.997682 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996344    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2435a2d-273d-4891-918d-377147d3eb63-cni-binary-copy\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.997682 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996370    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-874zc\" (UniqueName: \"kubernetes.io/projected/a2435a2d-273d-4891-918d-377147d3eb63-kube-api-access-874zc\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.997682 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996397    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2435a2d-273d-4891-918d-377147d3eb63-multus-daemon-config\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.997682 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996402    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ec0d0ea-61c0-4df1-9eb0-a78662baca1d-konnectivity-ca\") pod \"konnectivity-agent-jxqqz\" (UID: \"4ec0d0ea-61c0-4df1-9eb0-a78662baca1d\") " pod="kube-system/konnectivity-agent-jxqqz"
May 28 00:11:22.997682 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5d293d3-caa4-44cf-bae5-219a9e493213-cni-binary-copy\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:22.997682 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996464    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-kubelet\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.997682 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-cni-netd\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.997682 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996532    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/39bb9abb-9e4b-4adb-a92c-cb7c18563333-ovnkube-script-lib\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:22.997682 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996757    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/637d10d6-892a-46db-ac00-41e0294451d3-socket-dir\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:22.997682 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996826    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-host-run-multus-certs\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.997682 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996857    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2435a2d-273d-4891-918d-377147d3eb63-hostroot\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.997682 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996915    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-modprobe-d\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.997682 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.996922    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-sysconfig\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.997682 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.997438    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2435a2d-273d-4891-918d-377147d3eb63-cni-binary-copy\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:22.998322 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.998192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-etc-tuned\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:22.998841 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:22.998818    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ec0d0ea-61c0-4df1-9eb0-a78662baca1d-konnectivity-ca\") pod \"konnectivity-agent-jxqqz\" (UID: \"4ec0d0ea-61c0-4df1-9eb0-a78662baca1d\") " pod="kube-system/konnectivity-agent-jxqqz"
May 28 00:11:23.000228 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.000211    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-tmp\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:23.002183 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.002156    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ec0d0ea-61c0-4df1-9eb0-a78662baca1d-agent-certs\") pod \"konnectivity-agent-jxqqz\" (UID: \"4ec0d0ea-61c0-4df1-9eb0-a78662baca1d\") " pod="kube-system/konnectivity-agent-jxqqz"
May 28 00:11:23.026167 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:23.026137    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 00:11:23.026167 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:23.026169    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 00:11:23.026370 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:23.026184    2576 projected.go:194] Error preparing data for projected volume kube-api-access-kt654 for pod openshift-network-diagnostics/network-check-target-xq6dj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:23.026370 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:23.026244    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654 podName:2e67c0ae-9c40-46d4-91a8-a5577a46bd2f nodeName:}" failed. No retries permitted until 2026-05-28 00:11:23.526226599 +0000 UTC m=+3.147258591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kt654" (UniqueName: "kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654") pod "network-check-target-xq6dj" (UID: "2e67c0ae-9c40-46d4-91a8-a5577a46bd2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:23.028409 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.028373    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmbbg\" (UniqueName: \"kubernetes.io/projected/653f150e-1c4b-4dc5-a088-6ab0ad385350-kube-api-access-vmbbg\") pod \"network-metrics-daemon-6b84s\" (UID: \"653f150e-1c4b-4dc5-a088-6ab0ad385350\") " pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:23.029331 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.029306    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-874zc\" (UniqueName: \"kubernetes.io/projected/a2435a2d-273d-4891-918d-377147d3eb63-kube-api-access-874zc\") pod \"multus-62mcg\" (UID: \"a2435a2d-273d-4891-918d-377147d3eb63\") " pod="openshift-multus/multus-62mcg"
May 28 00:11:23.030935 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.030730    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 00:06:22 +0000 UTC" deadline="2028-03-05 15:37:19.564533382 +0000 UTC"
May 28 00:11:23.030935 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.030758    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15543h25m56.533778962s"
May 28 00:11:23.030935 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.030881    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86gdr\" (UniqueName: \"kubernetes.io/projected/d74ab08c-2275-4054-9365-ba62ea412c17-kube-api-access-86gdr\") pod \"node-resolver-tmc88\" (UID: \"d74ab08c-2275-4054-9365-ba62ea412c17\") " pod="openshift-dns/node-resolver-tmc88"
May 28 00:11:23.031185 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.031071    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqzwc\" (UniqueName: \"kubernetes.io/projected/a2f133ad-ace2-417a-a450-7f917265d112-kube-api-access-wqzwc\") pod \"node-ca-mvfbv\" (UID: \"a2f133ad-ace2-417a-a450-7f917265d112\") " pod="openshift-image-registry/node-ca-mvfbv"
May 28 00:11:23.031185 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.031097    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vn5z\" (UniqueName: \"kubernetes.io/projected/637d10d6-892a-46db-ac00-41e0294451d3-kube-api-access-6vn5z\") pod \"aws-ebs-csi-driver-node-hsfhl\" (UID: \"637d10d6-892a-46db-ac00-41e0294451d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:23.032902 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.032882    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjbpc\" (UniqueName: \"kubernetes.io/projected/6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88-kube-api-access-xjbpc\") pod \"tuned-zwmf5\" (UID: \"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88\") " pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:23.097620 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097537    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-run-ovn\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.097620 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097575    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5d293d3-caa4-44cf-bae5-219a9e493213-cnibin\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.097620 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097599    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-cni-bin\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.097871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097624    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5d293d3-caa4-44cf-bae5-219a9e493213-cni-binary-copy\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.097871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097645    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-kubelet\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.097871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097649    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-run-ovn\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.097871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097667    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-cni-netd\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.097871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097689    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/39bb9abb-9e4b-4adb-a92c-cb7c18563333-ovnkube-script-lib\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.097871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097701    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-cni-bin\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.097871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097712    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5d293d3-caa4-44cf-bae5-219a9e493213-os-release\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.097871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097741    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d293d3-caa4-44cf-bae5-219a9e493213-system-cni-dir\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.097871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097744    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5d293d3-caa4-44cf-bae5-219a9e493213-cnibin\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.097871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097765    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-var-lib-openvswitch\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.097871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097782    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-cni-netd\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.097871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097808    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cw887\" (UniqueName: \"kubernetes.io/projected/bd2bcc29-a5ec-4d13-832d-f74b151a4c3d-kube-api-access-cw887\") pod \"iptables-alerter-bvcpc\" (UID: \"bd2bcc29-a5ec-4d13-832d-f74b151a4c3d\") " pod="openshift-network-operator/iptables-alerter-bvcpc"
May 28 00:11:23.097871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097834    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-run-openvswitch\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.097871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097867    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097896    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/39bb9abb-9e4b-4adb-a92c-cb7c18563333-ovnkube-config\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097922    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/39bb9abb-9e4b-4adb-a92c-cb7c18563333-ovn-node-metrics-cert\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097948    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-slash\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.097974    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-log-socket\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098000    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qcjzk\" (UniqueName: \"kubernetes.io/projected/39bb9abb-9e4b-4adb-a92c-cb7c18563333-kube-api-access-qcjzk\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098029    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5d293d3-caa4-44cf-bae5-219a9e493213-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098059    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-run-systemd\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098088    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd2bcc29-a5ec-4d13-832d-f74b151a4c3d-host-slash\") pod \"iptables-alerter-bvcpc\" (UID: \"bd2bcc29-a5ec-4d13-832d-f74b151a4c3d\") " pod="openshift-network-operator/iptables-alerter-bvcpc"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098117    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/39bb9abb-9e4b-4adb-a92c-cb7c18563333-env-overrides\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098157    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-systemd-units\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098192    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-run-netns\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098217    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-node-log\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098249    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5d293d3-caa4-44cf-bae5-219a9e493213-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098278    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5d293d3-caa4-44cf-bae5-219a9e493213-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098284    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-kubelet\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098305    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-run-ovn-kubernetes\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.098492 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098332    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-log-socket\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.099269 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098333    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-etc-openvswitch\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.099269 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098380    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd2bcc29-a5ec-4d13-832d-f74b151a4c3d-iptables-alerter-script\") pod \"iptables-alerter-bvcpc\" (UID: \"bd2bcc29-a5ec-4d13-832d-f74b151a4c3d\") " pod="openshift-network-operator/iptables-alerter-bvcpc"
May 28 00:11:23.099269 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098415    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrzf\" (UniqueName: \"kubernetes.io/projected/d5d293d3-caa4-44cf-bae5-219a9e493213-kube-api-access-tjrzf\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.099269 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098695    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-run-netns\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.099269 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098245    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5d293d3-caa4-44cf-bae5-219a9e493213-cni-binary-copy\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.099269 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098751    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-systemd-units\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.099269 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098769    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-run-systemd\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.099269 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098794    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-node-log\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.099269 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098817    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-run-ovn-kubernetes\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.099269 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098851    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd2bcc29-a5ec-4d13-832d-f74b151a4c3d-host-slash\") pod \"iptables-alerter-bvcpc\" (UID: \"bd2bcc29-a5ec-4d13-832d-f74b151a4c3d\") " pod="openshift-network-operator/iptables-alerter-bvcpc"
May 28 00:11:23.099269 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.099199    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.099269 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.099219    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5d293d3-caa4-44cf-bae5-219a9e493213-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.099269 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.099270    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5d293d3-caa4-44cf-bae5-219a9e493213-os-release\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.099940 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.099282    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-host-slash\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.099940 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.099321    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-var-lib-openvswitch\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.099940 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.099322    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/39bb9abb-9e4b-4adb-a92c-cb7c18563333-env-overrides\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.099940 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.099362    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d293d3-caa4-44cf-bae5-219a9e493213-system-cni-dir\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.099940 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.099459    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5d293d3-caa4-44cf-bae5-219a9e493213-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.099940 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.099601    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-run-openvswitch\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.099940 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.098383    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/39bb9abb-9e4b-4adb-a92c-cb7c18563333-etc-openvswitch\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.099940 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.099779    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/39bb9abb-9e4b-4adb-a92c-cb7c18563333-ovnkube-config\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.099940 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.099910    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/39bb9abb-9e4b-4adb-a92c-cb7c18563333-ovnkube-script-lib\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.100370 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.099949    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd2bcc29-a5ec-4d13-832d-f74b151a4c3d-iptables-alerter-script\") pod \"iptables-alerter-bvcpc\" (UID: \"bd2bcc29-a5ec-4d13-832d-f74b151a4c3d\") " pod="openshift-network-operator/iptables-alerter-bvcpc"
May 28 00:11:23.100370 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.100332    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5d293d3-caa4-44cf-bae5-219a9e493213-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.102529 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.102485    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/39bb9abb-9e4b-4adb-a92c-cb7c18563333-ovn-node-metrics-cert\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.112910 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.112864    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcjzk\" (UniqueName: \"kubernetes.io/projected/39bb9abb-9e4b-4adb-a92c-cb7c18563333-kube-api-access-qcjzk\") pod \"ovnkube-node-8gct7\" (UID: \"39bb9abb-9e4b-4adb-a92c-cb7c18563333\") " pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.113068 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.112952    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrzf\" (UniqueName: \"kubernetes.io/projected/d5d293d3-caa4-44cf-bae5-219a9e493213-kube-api-access-tjrzf\") pod \"multus-additional-cni-plugins-r9lmd\" (UID: \"d5d293d3-caa4-44cf-bae5-219a9e493213\") " pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.119733 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.119708    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw887\" (UniqueName: \"kubernetes.io/projected/bd2bcc29-a5ec-4d13-832d-f74b151a4c3d-kube-api-access-cw887\") pod \"iptables-alerter-bvcpc\" (UID: \"bd2bcc29-a5ec-4d13-832d-f74b151a4c3d\") " pod="openshift-network-operator/iptables-alerter-bvcpc"
May 28 00:11:23.185433 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.185396    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl"
May 28 00:11:23.194558 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.194534    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zwmf5"
May 28 00:11:23.204186 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.204164    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tmc88"
May 28 00:11:23.209144 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.209120    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mvfbv"
May 28 00:11:23.216764 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.216746    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-62mcg"
May 28 00:11:23.225342 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.225322    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jxqqz"
May 28 00:11:23.231925 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.231896    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r9lmd"
May 28 00:11:23.240550 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.240527    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bvcpc"
May 28 00:11:23.247254 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.247232    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:23.329269 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.329237    2576 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 00:11:23.502599 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.502533    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs\") pod \"network-metrics-daemon-6b84s\" (UID: \"653f150e-1c4b-4dc5-a088-6ab0ad385350\") " pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:23.502727 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:23.502659    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:23.502727 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:23.502711    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs podName:653f150e-1c4b-4dc5-a088-6ab0ad385350 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:24.502699013 +0000 UTC m=+4.123731009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs") pod "network-metrics-daemon-6b84s" (UID: "653f150e-1c4b-4dc5-a088-6ab0ad385350") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:23.603793 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:23.603764    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kt654\" (UniqueName: \"kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654\") pod \"network-check-target-xq6dj\" (UID: \"2e67c0ae-9c40-46d4-91a8-a5577a46bd2f\") " pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:23.603969 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:23.603897    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 00:11:23.603969 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:23.603917    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 00:11:23.603969 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:23.603927    2576 projected.go:194] Error preparing data for projected volume kube-api-access-kt654 for pod openshift-network-diagnostics/network-check-target-xq6dj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:23.604122 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:23.603976    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654 podName:2e67c0ae-9c40-46d4-91a8-a5577a46bd2f nodeName:}" failed. No retries permitted until 2026-05-28 00:11:24.603961659 +0000 UTC m=+4.224993650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kt654" (UniqueName: "kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654") pod "network-check-target-xq6dj" (UID: "2e67c0ae-9c40-46d4-91a8-a5577a46bd2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:23.866343 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:23.866311    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637d10d6_892a_46db_ac00_41e0294451d3.slice/crio-3d92c482dda8a1fef5b0d861b1324465f88e81bf11664c29c71e7d2927554746 WatchSource:0}: Error finding container 3d92c482dda8a1fef5b0d861b1324465f88e81bf11664c29c71e7d2927554746: Status 404 returned error can't find the container with id 3d92c482dda8a1fef5b0d861b1324465f88e81bf11664c29c71e7d2927554746
May 28 00:11:23.867931 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:23.867801    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74ab08c_2275_4054_9365_ba62ea412c17.slice/crio-c33d5a24200fd73e97781f90dcbb4613e040bc9a9a3ac60237639378b2c5f6e8 WatchSource:0}: Error finding container c33d5a24200fd73e97781f90dcbb4613e040bc9a9a3ac60237639378b2c5f6e8: Status 404 returned error can't find the container with id c33d5a24200fd73e97781f90dcbb4613e040bc9a9a3ac60237639378b2c5f6e8
May 28 00:11:23.869688 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:23.869658    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d293d3_caa4_44cf_bae5_219a9e493213.slice/crio-3301afae95b8bfb09c70531a9534f2b05ac906003555fc621d5f69c2a39ba72c WatchSource:0}: Error finding container 3301afae95b8bfb09c70531a9534f2b05ac906003555fc621d5f69c2a39ba72c: Status 404 returned error can't find the container with id 3301afae95b8bfb09c70531a9534f2b05ac906003555fc621d5f69c2a39ba72c
May 28 00:11:23.871825 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:23.871802    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe0ac68_5a7f_48ae_b7ca_bc7ffc6ceb88.slice/crio-3444d91b97665d31ecc058ce2eabb456675d05da352c6831d0cce2554df3d3f0 WatchSource:0}: Error finding container 3444d91b97665d31ecc058ce2eabb456675d05da352c6831d0cce2554df3d3f0: Status 404 returned error can't find the container with id 3444d91b97665d31ecc058ce2eabb456675d05da352c6831d0cce2554df3d3f0
May 28 00:11:23.873493 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:23.873428    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec0d0ea_61c0_4df1_9eb0_a78662baca1d.slice/crio-c070e8ebb14bfe8f8f056ed990f78ad33cefb4958e2d3d8b1322d9078d861c1e WatchSource:0}: Error finding container c070e8ebb14bfe8f8f056ed990f78ad33cefb4958e2d3d8b1322d9078d861c1e: Status 404 returned error can't find the container with id c070e8ebb14bfe8f8f056ed990f78ad33cefb4958e2d3d8b1322d9078d861c1e
May 28 00:11:23.875529 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:23.875484    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f133ad_ace2_417a_a450_7f917265d112.slice/crio-55ddea3f0dfada9e002226cd5b3946fa215968bf452322f367830e77063d975e WatchSource:0}: Error finding container 55ddea3f0dfada9e002226cd5b3946fa215968bf452322f367830e77063d975e: Status 404 returned error can't find the container with id 55ddea3f0dfada9e002226cd5b3946fa215968bf452322f367830e77063d975e
May 28 00:11:23.876240 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:23.876219    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39bb9abb_9e4b_4adb_a92c_cb7c18563333.slice/crio-0a8c765f8d676711959ba428fe552eb6ab1351c221f7ef250dbd3453423a2b4e WatchSource:0}: Error finding container 0a8c765f8d676711959ba428fe552eb6ab1351c221f7ef250dbd3453423a2b4e: Status 404 returned error can't find the container with id 0a8c765f8d676711959ba428fe552eb6ab1351c221f7ef250dbd3453423a2b4e
May 28 00:11:23.877807 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:23.877775    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2435a2d_273d_4891_918d_377147d3eb63.slice/crio-406942ff992583a2bc63392e3c59351d5318fba22adf3348393ba2aff5011b2d WatchSource:0}: Error finding container 406942ff992583a2bc63392e3c59351d5318fba22adf3348393ba2aff5011b2d: Status 404 returned error can't find the container with id 406942ff992583a2bc63392e3c59351d5318fba22adf3348393ba2aff5011b2d
May 28 00:11:23.880205 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:11:23.880081    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2bcc29_a5ec_4d13_832d_f74b151a4c3d.slice/crio-3cdd69965747cc693d2cc4fcbd35a786978f8a9eadac840121392472ba8a4051 WatchSource:0}: Error finding container 3cdd69965747cc693d2cc4fcbd35a786978f8a9eadac840121392472ba8a4051: Status 404 returned error can't find the container with id 3cdd69965747cc693d2cc4fcbd35a786978f8a9eadac840121392472ba8a4051
May 28 00:11:24.019448 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:24.019414    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mvfbv" event={"ID":"a2f133ad-ace2-417a-a450-7f917265d112","Type":"ContainerStarted","Data":"55ddea3f0dfada9e002226cd5b3946fa215968bf452322f367830e77063d975e"}
May 28 00:11:24.020443 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:24.020411    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl" event={"ID":"637d10d6-892a-46db-ac00-41e0294451d3","Type":"ContainerStarted","Data":"3d92c482dda8a1fef5b0d861b1324465f88e81bf11664c29c71e7d2927554746"}
May 28 00:11:24.021491 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:24.021462    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bvcpc" event={"ID":"bd2bcc29-a5ec-4d13-832d-f74b151a4c3d","Type":"ContainerStarted","Data":"3cdd69965747cc693d2cc4fcbd35a786978f8a9eadac840121392472ba8a4051"}
May 28 00:11:24.022425 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:24.022400    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jxqqz" event={"ID":"4ec0d0ea-61c0-4df1-9eb0-a78662baca1d","Type":"ContainerStarted","Data":"c070e8ebb14bfe8f8f056ed990f78ad33cefb4958e2d3d8b1322d9078d861c1e"}
May 28 00:11:24.023367 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:24.023343    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zwmf5" event={"ID":"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88","Type":"ContainerStarted","Data":"3444d91b97665d31ecc058ce2eabb456675d05da352c6831d0cce2554df3d3f0"}
May 28 00:11:24.024343 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:24.024321    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9lmd" event={"ID":"d5d293d3-caa4-44cf-bae5-219a9e493213","Type":"ContainerStarted","Data":"3301afae95b8bfb09c70531a9534f2b05ac906003555fc621d5f69c2a39ba72c"}
May 28 00:11:24.025390 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:24.025363    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tmc88" event={"ID":"d74ab08c-2275-4054-9365-ba62ea412c17","Type":"ContainerStarted","Data":"c33d5a24200fd73e97781f90dcbb4613e040bc9a9a3ac60237639378b2c5f6e8"}
May 28 00:11:24.026920 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:24.026878    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal" event={"ID":"276f8146c60adf430fac9496a6d82374","Type":"ContainerStarted","Data":"f7178cff010b5dfb94127f7ca640bff15855823913e45113952a9011477a015c"}
May 28 00:11:24.028026 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:24.027997    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-62mcg" event={"ID":"a2435a2d-273d-4891-918d-377147d3eb63","Type":"ContainerStarted","Data":"406942ff992583a2bc63392e3c59351d5318fba22adf3348393ba2aff5011b2d"}
May 28 00:11:24.028912 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:24.028894    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7" event={"ID":"39bb9abb-9e4b-4adb-a92c-cb7c18563333","Type":"ContainerStarted","Data":"0a8c765f8d676711959ba428fe552eb6ab1351c221f7ef250dbd3453423a2b4e"}
May 28 00:11:24.030866 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:24.030838    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 00:06:22 +0000 UTC" deadline="2028-02-04 03:44:55.53445395 +0000 UTC"
May 28 00:11:24.030940 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:24.030867    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14811h33m31.503590212s"
May 28 00:11:24.042754 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:24.042716    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-163.ec2.internal" podStartSLOduration=2.042705235 podStartE2EDuration="2.042705235s" podCreationTimestamp="2026-05-28 00:11:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:11:24.042376125 +0000 UTC m=+3.663408138" watchObservedRunningTime="2026-05-28 00:11:24.042705235 +0000 UTC m=+3.663737259"
May 28 00:11:24.509257 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:24.509159    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs\") pod \"network-metrics-daemon-6b84s\" (UID: \"653f150e-1c4b-4dc5-a088-6ab0ad385350\") " pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:24.509450 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:24.509332    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:24.509450 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:24.509420    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs podName:653f150e-1c4b-4dc5-a088-6ab0ad385350 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:26.509398762 +0000 UTC m=+6.130430778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs") pod "network-metrics-daemon-6b84s" (UID: "653f150e-1c4b-4dc5-a088-6ab0ad385350") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:24.610923 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:24.610288    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kt654\" (UniqueName: \"kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654\") pod \"network-check-target-xq6dj\" (UID: \"2e67c0ae-9c40-46d4-91a8-a5577a46bd2f\") " pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:24.610923 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:24.610461    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 00:11:24.610923 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:24.610481    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 00:11:24.610923 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:24.610495    2576 projected.go:194] Error preparing data for projected volume kube-api-access-kt654 for pod openshift-network-diagnostics/network-check-target-xq6dj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:24.610923 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:24.610574    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654 podName:2e67c0ae-9c40-46d4-91a8-a5577a46bd2f nodeName:}" failed. No retries permitted until 2026-05-28 00:11:26.610552786 +0000 UTC m=+6.231584797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kt654" (UniqueName: "kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654") pod "network-check-target-xq6dj" (UID: "2e67c0ae-9c40-46d4-91a8-a5577a46bd2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:25.013665 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:25.012808    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:25.013665 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:25.012941    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:25.014317 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:25.014036    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:25.014317 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:25.014150    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:25.043446 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:25.043409    2576 generic.go:358] "Generic (PLEG): container finished" podID="2c3a83805b8518efc6135ad03a3963fc" containerID="132e6897e86df082382744ebedd8b415447a9a98f912d1f1110ca56f5c31fcc4" exitCode=0
May 28 00:11:25.044282 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:25.044258    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal" event={"ID":"2c3a83805b8518efc6135ad03a3963fc","Type":"ContainerDied","Data":"132e6897e86df082382744ebedd8b415447a9a98f912d1f1110ca56f5c31fcc4"}
May 28 00:11:26.051922 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:26.051887    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal" event={"ID":"2c3a83805b8518efc6135ad03a3963fc","Type":"ContainerStarted","Data":"75f37c9c5d6cc149807b0ed07bca8c4f975a03fadb75ce1f7c863ff0e6f5d962"}
May 28 00:11:26.526235 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:26.525646    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs\") pod \"network-metrics-daemon-6b84s\" (UID: \"653f150e-1c4b-4dc5-a088-6ab0ad385350\") " pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:26.526235 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:26.525778    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:26.526235 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:26.525841    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs podName:653f150e-1c4b-4dc5-a088-6ab0ad385350 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:30.525822 +0000 UTC m=+10.146854011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs") pod "network-metrics-daemon-6b84s" (UID: "653f150e-1c4b-4dc5-a088-6ab0ad385350") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:26.627376 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:26.626759    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kt654\" (UniqueName: \"kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654\") pod \"network-check-target-xq6dj\" (UID: \"2e67c0ae-9c40-46d4-91a8-a5577a46bd2f\") " pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:26.627376 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:26.626921    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 00:11:26.627376 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:26.626941    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 00:11:26.627376 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:26.626954    2576 projected.go:194] Error preparing data for projected volume kube-api-access-kt654 for pod openshift-network-diagnostics/network-check-target-xq6dj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:26.627376 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:26.627017    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654 podName:2e67c0ae-9c40-46d4-91a8-a5577a46bd2f nodeName:}" failed. No retries permitted until 2026-05-28 00:11:30.626998173 +0000 UTC m=+10.248030164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kt654" (UniqueName: "kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654") pod "network-check-target-xq6dj" (UID: "2e67c0ae-9c40-46d4-91a8-a5577a46bd2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:27.013292 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:27.012640    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:27.013292 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:27.012784    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:27.016047 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:27.013873    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:27.016047 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:27.014032    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:28.140329 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:28.140264    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-163.ec2.internal" podStartSLOduration=6.140245807 podStartE2EDuration="6.140245807s" podCreationTimestamp="2026-05-28 00:11:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:11:26.069141017 +0000 UTC m=+5.690173032" watchObservedRunningTime="2026-05-28 00:11:28.140245807 +0000 UTC m=+7.761277822"
May 28 00:11:28.141747 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:28.141271    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-h7v4h"]
May 28 00:11:28.146020 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:28.145769    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:28.146020 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:28.145857    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7v4h" podUID="3a977914-7378-4daf-a7ca-b1e54548a760"
May 28 00:11:28.240889 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:28.240849    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret\") pod \"global-pull-secret-syncer-h7v4h\" (UID: \"3a977914-7378-4daf-a7ca-b1e54548a760\") " pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:28.241076 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:28.240930    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a977914-7378-4daf-a7ca-b1e54548a760-dbus\") pod \"global-pull-secret-syncer-h7v4h\" (UID: \"3a977914-7378-4daf-a7ca-b1e54548a760\") " pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:28.241076 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:28.240986    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a977914-7378-4daf-a7ca-b1e54548a760-kubelet-config\") pod \"global-pull-secret-syncer-h7v4h\" (UID: \"3a977914-7378-4daf-a7ca-b1e54548a760\") " pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:28.341985 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:28.341944    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret\") pod \"global-pull-secret-syncer-h7v4h\" (UID: \"3a977914-7378-4daf-a7ca-b1e54548a760\") " pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:28.342166 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:28.342021    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a977914-7378-4daf-a7ca-b1e54548a760-dbus\") pod \"global-pull-secret-syncer-h7v4h\" (UID: \"3a977914-7378-4daf-a7ca-b1e54548a760\") " pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:28.342166 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:28.342077    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a977914-7378-4daf-a7ca-b1e54548a760-kubelet-config\") pod \"global-pull-secret-syncer-h7v4h\" (UID: \"3a977914-7378-4daf-a7ca-b1e54548a760\") " pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:28.342279 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:28.342169    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a977914-7378-4daf-a7ca-b1e54548a760-kubelet-config\") pod \"global-pull-secret-syncer-h7v4h\" (UID: \"3a977914-7378-4daf-a7ca-b1e54548a760\") " pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:28.342333 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:28.342287    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 00:11:28.342382 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:28.342352    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret podName:3a977914-7378-4daf-a7ca-b1e54548a760 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:28.842332116 +0000 UTC m=+8.463364130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret") pod "global-pull-secret-syncer-h7v4h" (UID: "3a977914-7378-4daf-a7ca-b1e54548a760") : object "kube-system"/"original-pull-secret" not registered
May 28 00:11:28.342853 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:28.342759    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a977914-7378-4daf-a7ca-b1e54548a760-dbus\") pod \"global-pull-secret-syncer-h7v4h\" (UID: \"3a977914-7378-4daf-a7ca-b1e54548a760\") " pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:28.847338 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:28.847042    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret\") pod \"global-pull-secret-syncer-h7v4h\" (UID: \"3a977914-7378-4daf-a7ca-b1e54548a760\") " pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:28.847338 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:28.847181    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 00:11:28.847338 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:28.847245    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret podName:3a977914-7378-4daf-a7ca-b1e54548a760 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:29.847227139 +0000 UTC m=+9.468259132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret") pod "global-pull-secret-syncer-h7v4h" (UID: "3a977914-7378-4daf-a7ca-b1e54548a760") : object "kube-system"/"original-pull-secret" not registered
May 28 00:11:29.012522 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:29.012278    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:29.012522 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:29.012313    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:29.012522 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:29.012405    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:29.012828 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:29.012789    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:29.856053 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:29.855968    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret\") pod \"global-pull-secret-syncer-h7v4h\" (UID: \"3a977914-7378-4daf-a7ca-b1e54548a760\") " pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:29.856480 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:29.856119    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 00:11:29.856480 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:29.856194    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret podName:3a977914-7378-4daf-a7ca-b1e54548a760 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:31.856174205 +0000 UTC m=+11.477206200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret") pod "global-pull-secret-syncer-h7v4h" (UID: "3a977914-7378-4daf-a7ca-b1e54548a760") : object "kube-system"/"original-pull-secret" not registered
May 28 00:11:30.012077 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:30.012040    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:30.012263 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:30.012163    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7v4h" podUID="3a977914-7378-4daf-a7ca-b1e54548a760"
May 28 00:11:30.560733 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:30.560694    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs\") pod \"network-metrics-daemon-6b84s\" (UID: \"653f150e-1c4b-4dc5-a088-6ab0ad385350\") " pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:30.560914 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:30.560845    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:30.560981 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:30.560917    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs podName:653f150e-1c4b-4dc5-a088-6ab0ad385350 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:38.560897218 +0000 UTC m=+18.181929212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs") pod "network-metrics-daemon-6b84s" (UID: "653f150e-1c4b-4dc5-a088-6ab0ad385350") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:30.662187 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:30.662147    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kt654\" (UniqueName: \"kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654\") pod \"network-check-target-xq6dj\" (UID: \"2e67c0ae-9c40-46d4-91a8-a5577a46bd2f\") " pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:30.662381 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:30.662309    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 00:11:30.662381 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:30.662334    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 00:11:30.662381 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:30.662350    2576 projected.go:194] Error preparing data for projected volume kube-api-access-kt654 for pod openshift-network-diagnostics/network-check-target-xq6dj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:30.662614 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:30.662428    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654 podName:2e67c0ae-9c40-46d4-91a8-a5577a46bd2f nodeName:}" failed. No retries permitted until 2026-05-28 00:11:38.662400475 +0000 UTC m=+18.283432476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kt654" (UniqueName: "kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654") pod "network-check-target-xq6dj" (UID: "2e67c0ae-9c40-46d4-91a8-a5577a46bd2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:31.012968 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:31.012877    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:31.013434 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:31.013008    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:31.013434 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:31.013367    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:31.013573 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:31.013453    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:31.872421 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:31.872333    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret\") pod \"global-pull-secret-syncer-h7v4h\" (UID: \"3a977914-7378-4daf-a7ca-b1e54548a760\") " pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:31.872612 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:31.872500    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 00:11:31.872612 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:31.872601    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret podName:3a977914-7378-4daf-a7ca-b1e54548a760 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:35.872580141 +0000 UTC m=+15.493612138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret") pod "global-pull-secret-syncer-h7v4h" (UID: "3a977914-7378-4daf-a7ca-b1e54548a760") : object "kube-system"/"original-pull-secret" not registered
May 28 00:11:32.012155 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:32.012037    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:32.012339 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:32.012173    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7v4h" podUID="3a977914-7378-4daf-a7ca-b1e54548a760"
May 28 00:11:33.012932 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:33.012429    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:33.012932 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:33.012576    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:33.012932 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:33.012619    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:33.012932 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:33.012740    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:34.012239 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:34.012197    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:34.012419 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:34.012331    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7v4h" podUID="3a977914-7378-4daf-a7ca-b1e54548a760"
May 28 00:11:35.012342 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:35.012307    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:35.012844 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:35.012306    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:35.012844 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:35.012456    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:35.012844 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:35.012558    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:35.902705 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:35.902660    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret\") pod \"global-pull-secret-syncer-h7v4h\" (UID: \"3a977914-7378-4daf-a7ca-b1e54548a760\") " pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:35.902900 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:35.902816    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 00:11:35.902900 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:35.902888    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret podName:3a977914-7378-4daf-a7ca-b1e54548a760 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:43.902869535 +0000 UTC m=+23.523901529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret") pod "global-pull-secret-syncer-h7v4h" (UID: "3a977914-7378-4daf-a7ca-b1e54548a760") : object "kube-system"/"original-pull-secret" not registered
May 28 00:11:36.011715 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:36.011677    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:36.011886 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:36.011805    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7v4h" podUID="3a977914-7378-4daf-a7ca-b1e54548a760"
May 28 00:11:37.011655 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:37.011621    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:37.012104 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:37.011620    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:37.012104 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:37.011754    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:37.012104 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:37.011856    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:38.012301 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:38.012262    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:38.012842 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:38.012392    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7v4h" podUID="3a977914-7378-4daf-a7ca-b1e54548a760"
May 28 00:11:38.623012 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:38.622972    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs\") pod \"network-metrics-daemon-6b84s\" (UID: \"653f150e-1c4b-4dc5-a088-6ab0ad385350\") " pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:38.623164 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:38.623150    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:38.623245 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:38.623226    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs podName:653f150e-1c4b-4dc5-a088-6ab0ad385350 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:54.623205914 +0000 UTC m=+34.244237910 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs") pod "network-metrics-daemon-6b84s" (UID: "653f150e-1c4b-4dc5-a088-6ab0ad385350") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:38.723755 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:38.723719    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kt654\" (UniqueName: \"kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654\") pod \"network-check-target-xq6dj\" (UID: \"2e67c0ae-9c40-46d4-91a8-a5577a46bd2f\") " pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:38.723922 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:38.723887    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 00:11:38.723922 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:38.723912    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 00:11:38.723922 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:38.723922    2576 projected.go:194] Error preparing data for projected volume kube-api-access-kt654 for pod openshift-network-diagnostics/network-check-target-xq6dj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:38.724031 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:38.723985    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654 podName:2e67c0ae-9c40-46d4-91a8-a5577a46bd2f nodeName:}" failed. No retries permitted until 2026-05-28 00:11:54.723970978 +0000 UTC m=+34.345002969 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kt654" (UniqueName: "kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654") pod "network-check-target-xq6dj" (UID: "2e67c0ae-9c40-46d4-91a8-a5577a46bd2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:39.012108 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:39.012020    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:39.012108 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:39.012062    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:39.012310 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:39.012146    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:39.012310 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:39.012279    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:40.012491 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:40.012407    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:40.012961 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:40.012581    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7v4h" podUID="3a977914-7378-4daf-a7ca-b1e54548a760"
May 28 00:11:41.012964 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:41.012759    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:41.013360 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:41.012827    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:41.013360 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:41.013039    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:41.013360 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:41.013114    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:42.011677 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:42.011647    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:42.011785 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:42.011757    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7v4h" podUID="3a977914-7378-4daf-a7ca-b1e54548a760"
May 28 00:11:43.012402 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.012363    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:43.012402 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.012390    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:43.013295 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:43.012554    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:43.013295 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:43.012666    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:43.081960 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.081926    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-62mcg" event={"ID":"a2435a2d-273d-4891-918d-377147d3eb63","Type":"ContainerStarted","Data":"99742575f7c5637363d7c06dc91b60f638066a5cfa2c7ce8107cebfe65cd2d2a"}
May 28 00:11:43.085247 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.085214    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7" event={"ID":"39bb9abb-9e4b-4adb-a92c-cb7c18563333","Type":"ContainerStarted","Data":"5de0afc19101741bae94c0cfc529c1ed4fb69d933f0022c0c011db6fc584189e"}
May 28 00:11:43.085364 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.085255    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7" event={"ID":"39bb9abb-9e4b-4adb-a92c-cb7c18563333","Type":"ContainerStarted","Data":"970ae030b925064f857c8a3c52fe5df2fe1a56e6892b01065870e7e05bde0530"}
May 28 00:11:43.085364 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.085264    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7" event={"ID":"39bb9abb-9e4b-4adb-a92c-cb7c18563333","Type":"ContainerStarted","Data":"a1e7b65ac84f61cda721aa67c806e7a24eda694f05797f01160a9bb619962fe5"}
May 28 00:11:43.085364 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.085273    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7" event={"ID":"39bb9abb-9e4b-4adb-a92c-cb7c18563333","Type":"ContainerStarted","Data":"2800b251a0642062bdba49d04d7ff28b27fa55b605e7809d5ab182b0bffe3d74"}
May 28 00:11:43.085364 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.085280    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7" event={"ID":"39bb9abb-9e4b-4adb-a92c-cb7c18563333","Type":"ContainerStarted","Data":"d811ce64fb8e54c53ef049b20853a4956fcaf333b597e20c7f56498d9075d917"}
May 28 00:11:43.085364 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.085288    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7" event={"ID":"39bb9abb-9e4b-4adb-a92c-cb7c18563333","Type":"ContainerStarted","Data":"7e6f130dc92f983cb3a283426f0b7eeaae88ae4be05531a4f207f3a47b262115"}
May 28 00:11:43.087261 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.087232    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mvfbv" event={"ID":"a2f133ad-ace2-417a-a450-7f917265d112","Type":"ContainerStarted","Data":"b31a49cff1c5c68a24c6e144ee7b15fd54113fb09b97da851b669df29ceaa4e2"}
May 28 00:11:43.089418 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.089391    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl" event={"ID":"637d10d6-892a-46db-ac00-41e0294451d3","Type":"ContainerStarted","Data":"23de49ec46b8fb1eb9150c057c2c9063fd7520ef31a1ce9eb989f4728e642616"}
May 28 00:11:43.091177 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.091146    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jxqqz" event={"ID":"4ec0d0ea-61c0-4df1-9eb0-a78662baca1d","Type":"ContainerStarted","Data":"82dac520d1e16d2dc24781ded50be46a9e969a05485bdbd8b0ac8e177d1aa8c5"}
May 28 00:11:43.092919 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.092891    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zwmf5" event={"ID":"6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88","Type":"ContainerStarted","Data":"3f6b451b4b3cace21602bce32be90ee65e5464c56047aef896b1abcdaa679c6f"}
May 28 00:11:43.094586 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.094559    2576 generic.go:358] "Generic (PLEG): container finished" podID="d5d293d3-caa4-44cf-bae5-219a9e493213" containerID="fee226622c8ea0ea8814e4d0cd64cda4eedc660fd8aa52d135f34d6b51d07a54" exitCode=0
May 28 00:11:43.094685 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.094631    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9lmd" event={"ID":"d5d293d3-caa4-44cf-bae5-219a9e493213","Type":"ContainerDied","Data":"fee226622c8ea0ea8814e4d0cd64cda4eedc660fd8aa52d135f34d6b51d07a54"}
May 28 00:11:43.096296 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.096274    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tmc88" event={"ID":"d74ab08c-2275-4054-9365-ba62ea412c17","Type":"ContainerStarted","Data":"86013d65b24da851810e55bddd90cb77bef8bc0c643d451ccc7f2d54873ee301"}
May 28 00:11:43.114694 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.114651    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mvfbv" podStartSLOduration=3.7879995539999998 podStartE2EDuration="22.114635349s" podCreationTimestamp="2026-05-28 00:11:21 +0000 UTC" firstStartedPulling="2026-05-28 00:11:23.878412937 +0000 UTC m=+3.499444929" lastFinishedPulling="2026-05-28 00:11:42.205048719 +0000 UTC m=+21.826080724" observedRunningTime="2026-05-28 00:11:43.114521336 +0000 UTC m=+22.735553343" watchObservedRunningTime="2026-05-28 00:11:43.114635349 +0000 UTC m=+22.735667361"
May 28 00:11:43.114941 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.114910    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-62mcg" podStartSLOduration=3.773638306 podStartE2EDuration="22.114901742s" podCreationTimestamp="2026-05-28 00:11:21 +0000 UTC" firstStartedPulling="2026-05-28 00:11:23.88037706 +0000 UTC m=+3.501409051" lastFinishedPulling="2026-05-28 00:11:42.221640492 +0000 UTC m=+21.842672487" observedRunningTime="2026-05-28 00:11:43.100569283 +0000 UTC m=+22.721601299" watchObservedRunningTime="2026-05-28 00:11:43.114901742 +0000 UTC m=+22.735933755"
May 28 00:11:43.130690 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.130643    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zwmf5" podStartSLOduration=3.798066571 podStartE2EDuration="22.130626816s" podCreationTimestamp="2026-05-28 00:11:21 +0000 UTC" firstStartedPulling="2026-05-28 00:11:23.873749753 +0000 UTC m=+3.494781757" lastFinishedPulling="2026-05-28 00:11:42.20631001 +0000 UTC m=+21.827342002" observedRunningTime="2026-05-28 00:11:43.130596561 +0000 UTC m=+22.751628575" watchObservedRunningTime="2026-05-28 00:11:43.130626816 +0000 UTC m=+22.751658862"
May 28 00:11:43.146588 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.146536    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-tmc88" podStartSLOduration=3.811538536 podStartE2EDuration="22.14651893s" podCreationTimestamp="2026-05-28 00:11:21 +0000 UTC" firstStartedPulling="2026-05-28 00:11:23.869921741 +0000 UTC m=+3.490953733" lastFinishedPulling="2026-05-28 00:11:42.204902121 +0000 UTC m=+21.825934127" observedRunningTime="2026-05-28 00:11:43.146351796 +0000 UTC m=+22.767383820" watchObservedRunningTime="2026-05-28 00:11:43.14651893 +0000 UTC m=+22.767550942"
May 28 00:11:43.195410 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.195357    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jxqqz" podStartSLOduration=8.814219683 podStartE2EDuration="22.195343155s" podCreationTimestamp="2026-05-28 00:11:21 +0000 UTC" firstStartedPulling="2026-05-28 00:11:23.877460431 +0000 UTC m=+3.498492426" lastFinishedPulling="2026-05-28 00:11:37.258583898 +0000 UTC m=+16.879615898" observedRunningTime="2026-05-28 00:11:43.16812474 +0000 UTC m=+22.789156753" watchObservedRunningTime="2026-05-28 00:11:43.195343155 +0000 UTC m=+22.816375151"
May 28 00:11:43.699172 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.698994    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 28 00:11:43.958640 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.958477    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-28T00:11:43.699154468Z","UUID":"707fd485-1f62-4619-869f-0432b4ec2289","Handler":null,"Name":"","Endpoint":""}
May 28 00:11:43.960159 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.960135    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 28 00:11:43.960314 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.960167    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 28 00:11:43.968635 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:43.968612    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret\") pod \"global-pull-secret-syncer-h7v4h\" (UID: \"3a977914-7378-4daf-a7ca-b1e54548a760\") " pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:43.968763 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:43.968740    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 00:11:43.968822 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:43.968812    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret podName:3a977914-7378-4daf-a7ca-b1e54548a760 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:59.96879284 +0000 UTC m=+39.589824833 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret") pod "global-pull-secret-syncer-h7v4h" (UID: "3a977914-7378-4daf-a7ca-b1e54548a760") : object "kube-system"/"original-pull-secret" not registered
May 28 00:11:44.012173 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:44.012143    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:44.012348 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:44.012268    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7v4h" podUID="3a977914-7378-4daf-a7ca-b1e54548a760"
May 28 00:11:44.099839 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:44.099802    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl" event={"ID":"637d10d6-892a-46db-ac00-41e0294451d3","Type":"ContainerStarted","Data":"3e3b82ee77b31cd96c64c6f4bb8f761f83a77553dc823aa1eed4763614a97671"}
May 28 00:11:44.101023 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:44.100993    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bvcpc" event={"ID":"bd2bcc29-a5ec-4d13-832d-f74b151a4c3d","Type":"ContainerStarted","Data":"32158f4076aafc4573f9931f9f1ab895c41d2bb26ca944832b2b61386236d075"}
May 28 00:11:44.116206 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:44.116162    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bvcpc" podStartSLOduration=9.739783467 podStartE2EDuration="23.116150242s" podCreationTimestamp="2026-05-28 00:11:21 +0000 UTC" firstStartedPulling="2026-05-28 00:11:23.882252171 +0000 UTC m=+3.503284162" lastFinishedPulling="2026-05-28 00:11:37.258618946 +0000 UTC m=+16.879650937" observedRunningTime="2026-05-28 00:11:44.11612586 +0000 UTC m=+23.737157872" watchObservedRunningTime="2026-05-28 00:11:44.116150242 +0000 UTC m=+23.737182254"
May 28 00:11:44.193539 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:44.193496    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jxqqz"
May 28 00:11:44.194100 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:44.194083    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jxqqz"
May 28 00:11:45.011549 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:45.011494    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:45.011664 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:45.011642    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:45.011723 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:45.011702    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:45.011820 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:45.011802    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:45.107287 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:45.107245    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7" event={"ID":"39bb9abb-9e4b-4adb-a92c-cb7c18563333","Type":"ContainerStarted","Data":"fe059d7a426be0391fa3b341c74896d4afdca1d5b020e649c7fb5e71c63185be"}
May 28 00:11:45.109500 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:45.109399    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl" event={"ID":"637d10d6-892a-46db-ac00-41e0294451d3","Type":"ContainerStarted","Data":"fc66a3203013281aa1dc43caa35c84d87d10246d763832a6571833c00cf927fc"}
May 28 00:11:45.109821 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:45.109798    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jxqqz"
May 28 00:11:45.110264 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:45.110243    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jxqqz"
May 28 00:11:45.128962 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:45.128913    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsfhl" podStartSLOduration=3.061766358 podStartE2EDuration="24.128899759s" podCreationTimestamp="2026-05-28 00:11:21 +0000 UTC" firstStartedPulling="2026-05-28 00:11:23.868165468 +0000 UTC m=+3.489197460" lastFinishedPulling="2026-05-28 00:11:44.935298867 +0000 UTC m=+24.556330861" observedRunningTime="2026-05-28 00:11:45.128281197 +0000 UTC m=+24.749313222" watchObservedRunningTime="2026-05-28 00:11:45.128899759 +0000 UTC m=+24.749931771"
May 28 00:11:46.011703 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:46.011671    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:46.011871 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:46.011776    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7v4h" podUID="3a977914-7378-4daf-a7ca-b1e54548a760"
May 28 00:11:47.011483 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:47.011401    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:47.011483 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:47.011401    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:47.012259 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:47.011552    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:47.012259 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:47.011559    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:48.011917 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:48.011663    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:48.012570 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:48.012015    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7v4h" podUID="3a977914-7378-4daf-a7ca-b1e54548a760"
May 28 00:11:48.118208 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:48.118172    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7" event={"ID":"39bb9abb-9e4b-4adb-a92c-cb7c18563333","Type":"ContainerStarted","Data":"51b489b72d76b1c0f212eb567b5428b9192a67b2c70495afe525b39d1684c647"}
May 28 00:11:48.119716 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:48.119665    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:48.119716 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:48.119717    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:48.138656 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:48.138630    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:48.138791 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:48.138740    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:48.151892 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:48.151839    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7" podStartSLOduration=8.402782608 podStartE2EDuration="27.15182311s" podCreationTimestamp="2026-05-28 00:11:21 +0000 UTC" firstStartedPulling="2026-05-28 00:11:23.879697669 +0000 UTC m=+3.500729670" lastFinishedPulling="2026-05-28 00:11:42.628738165 +0000 UTC m=+22.249770172" observedRunningTime="2026-05-28 00:11:48.151402184 +0000 UTC m=+27.772434195" watchObservedRunningTime="2026-05-28 00:11:48.15182311 +0000 UTC m=+27.772855104"
May 28 00:11:49.011635 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:49.011599    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:49.011635 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:49.011640    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:49.012438 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:49.011735    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:49.012438 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:49.011877    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:49.084986 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:49.084634    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h7v4h"]
May 28 00:11:49.084986 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:49.084784    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:49.084986 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:49.084898    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7v4h" podUID="3a977914-7378-4daf-a7ca-b1e54548a760"
May 28 00:11:49.087568 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:49.087545    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6b84s"]
May 28 00:11:49.099046 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:49.099019    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xq6dj"]
May 28 00:11:49.120144 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:49.120099    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:49.120301 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:49.120217    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:49.120301 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:49.120223    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:49.120418 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:49.120320    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:49.120480 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:49.120437    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 28 00:11:49.966720 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:49.966694    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:11:50.123803 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:50.123732    2576 generic.go:358] "Generic (PLEG): container finished" podID="d5d293d3-caa4-44cf-bae5-219a9e493213" containerID="5782a3ac68d788833c5472298adb66bf0ebea0a3300a55ee25a8b1c2ab568f8c" exitCode=0
May 28 00:11:50.124149 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:50.123815    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9lmd" event={"ID":"d5d293d3-caa4-44cf-bae5-219a9e493213","Type":"ContainerDied","Data":"5782a3ac68d788833c5472298adb66bf0ebea0a3300a55ee25a8b1c2ab568f8c"}
May 28 00:11:51.012549 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:51.012496    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:51.012729 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:51.012624    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:51.012729 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:51.012646    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:51.012729 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:51.012717    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7v4h" podUID="3a977914-7378-4daf-a7ca-b1e54548a760"
May 28 00:11:51.012893 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:51.012620    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:51.012893 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:51.012792    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:52.130248 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:52.130061    2576 generic.go:358] "Generic (PLEG): container finished" podID="d5d293d3-caa4-44cf-bae5-219a9e493213" containerID="3ad8a8481259091a65d88241e7ecf89ab0e1f5c271664f65cfe2910c1ce59392" exitCode=0
May 28 00:11:52.130248 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:52.130143    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9lmd" event={"ID":"d5d293d3-caa4-44cf-bae5-219a9e493213","Type":"ContainerDied","Data":"3ad8a8481259091a65d88241e7ecf89ab0e1f5c271664f65cfe2910c1ce59392"}
May 28 00:11:53.011742 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:53.011658    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:53.011742 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:53.011699    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:53.011742 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:53.011742    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:53.011949 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:53.011827    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7v4h" podUID="3a977914-7378-4daf-a7ca-b1e54548a760"
May 28 00:11:53.011990 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:53.011944    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:53.012068 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:53.012046    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:53.134421 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:53.134388    2576 generic.go:358] "Generic (PLEG): container finished" podID="d5d293d3-caa4-44cf-bae5-219a9e493213" containerID="b349cd43811715890a1b3375edf23e99b9c3cafb9e989c217ff3a6a4cfd80598" exitCode=0
May 28 00:11:53.134850 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:53.134451    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9lmd" event={"ID":"d5d293d3-caa4-44cf-bae5-219a9e493213","Type":"ContainerDied","Data":"b349cd43811715890a1b3375edf23e99b9c3cafb9e989c217ff3a6a4cfd80598"}
May 28 00:11:54.656810 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:54.656766    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs\") pod \"network-metrics-daemon-6b84s\" (UID: \"653f150e-1c4b-4dc5-a088-6ab0ad385350\") " pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:54.657269 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:54.656905    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:54.657269 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:54.656962    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs podName:653f150e-1c4b-4dc5-a088-6ab0ad385350 nodeName:}" failed. No retries permitted until 2026-05-28 00:12:26.656947599 +0000 UTC m=+66.277979595 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs") pod "network-metrics-daemon-6b84s" (UID: "653f150e-1c4b-4dc5-a088-6ab0ad385350") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 00:11:54.757124 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:54.757090    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kt654\" (UniqueName: \"kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654\") pod \"network-check-target-xq6dj\" (UID: \"2e67c0ae-9c40-46d4-91a8-a5577a46bd2f\") " pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:54.757309 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:54.757286    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 00:11:54.757362 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:54.757315    2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 00:11:54.757362 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:54.757330    2576 projected.go:194] Error preparing data for projected volume kube-api-access-kt654 for pod openshift-network-diagnostics/network-check-target-xq6dj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:54.757439 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:54.757396    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654 podName:2e67c0ae-9c40-46d4-91a8-a5577a46bd2f nodeName:}" failed. No retries permitted until 2026-05-28 00:12:26.757376987 +0000 UTC m=+66.378408984 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kt654" (UniqueName: "kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654") pod "network-check-target-xq6dj" (UID: "2e67c0ae-9c40-46d4-91a8-a5577a46bd2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 00:11:55.012259 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.012170    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:55.012259 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.012221    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:55.012486 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:55.012308    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h7v4h" podUID="3a977914-7378-4daf-a7ca-b1e54548a760"
May 28 00:11:55.012565 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:55.012488    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq6dj" podUID="2e67c0ae-9c40-46d4-91a8-a5577a46bd2f"
May 28 00:11:55.012565 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.012561    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:55.012659 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:55.012633    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:11:55.218385 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.218357    2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-163.ec2.internal" event="NodeReady"
May 28 00:11:55.218555 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.218517    2576 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 28 00:11:55.255650 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.255618    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79b94db5c6-b4nm5"]
May 28 00:11:55.257482 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.257462    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.259951 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.259928    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 28 00:11:55.260289 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.260272    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 28 00:11:55.260369 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.260290    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-srcnj\""
May 28 00:11:55.260369 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.260281    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 28 00:11:55.266820 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.266741    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 28 00:11:55.267325 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.267302    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4sq87"]
May 28 00:11:55.273047 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.273025    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tgwt7"]
May 28 00:11:55.273191 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.273173    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4sq87"
May 28 00:11:55.274572 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.274552    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79b94db5c6-b4nm5"]
May 28 00:11:55.274686 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.274645    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:11:55.275693 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.275558    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzrg4\""
May 28 00:11:55.275899 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.275885    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 28 00:11:55.275991 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.275921    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 28 00:11:55.277052 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.277016    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4sq87"]
May 28 00:11:55.277578 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.277232    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 28 00:11:55.277770 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.277724    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 28 00:11:55.277838 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.277771    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nx64p\""
May 28 00:11:55.278126 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.277966    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 28 00:11:55.283807 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.283787    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tgwt7"]
May 28 00:11:55.361602 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.361572    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-certificates\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.361762 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.361611    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a0045af-7f37-4ae7-89ec-7c66c799c21c-image-registry-private-configuration\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.361762 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.361640    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-bound-sa-token\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.361762 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.361666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t46r\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-kube-api-access-5t46r\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.361762 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.361750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.361951 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.361812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a0045af-7f37-4ae7-89ec-7c66c799c21c-trusted-ca\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.361951 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.361841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a0045af-7f37-4ae7-89ec-7c66c799c21c-installation-pull-secrets\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.361951 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.361873    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a0045af-7f37-4ae7-89ec-7c66c799c21c-ca-trust-extracted\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.462886 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.462844    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e2c7ab9-c05a-44d8-907a-377754c23b53-tmp-dir\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:11:55.462886 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.462891    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46zj2\" (UniqueName: \"kubernetes.io/projected/2e2c7ab9-c05a-44d8-907a-377754c23b53-kube-api-access-46zj2\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:11:55.463095 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.462949    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a0045af-7f37-4ae7-89ec-7c66c799c21c-trusted-ca\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.463095 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.463002    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a0045af-7f37-4ae7-89ec-7c66c799c21c-installation-pull-secrets\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.463095 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.463050    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a0045af-7f37-4ae7-89ec-7c66c799c21c-ca-trust-extracted\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.463095 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.463074    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert\") pod \"ingress-canary-tgwt7\" (UID: \"b1715227-4b2d-44dd-a153-51973245938a\") " pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:11:55.463281 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.463103    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qqjb\" (UniqueName: \"kubernetes.io/projected/b1715227-4b2d-44dd-a153-51973245938a-kube-api-access-8qqjb\") pod \"ingress-canary-tgwt7\" (UID: \"b1715227-4b2d-44dd-a153-51973245938a\") " pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:11:55.463281 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.463149    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-certificates\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.463281 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.463192    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a0045af-7f37-4ae7-89ec-7c66c799c21c-image-registry-private-configuration\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.463281 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.463219    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-bound-sa-token\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.463281 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.463245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2c7ab9-c05a-44d8-907a-377754c23b53-config-volume\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:11:55.463281 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.463274    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5t46r\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-kube-api-access-5t46r\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.463595 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.463316    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:11:55.463595 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.463347    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.463595 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:55.463446    2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 00:11:55.463595 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:55.463456    2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79b94db5c6-b4nm5: secret "image-registry-tls" not found
May 28 00:11:55.463595 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:55.463550    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls podName:0a0045af-7f37-4ae7-89ec-7c66c799c21c nodeName:}" failed. No retries permitted until 2026-05-28 00:11:55.96349096 +0000 UTC m=+35.584522951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls") pod "image-registry-79b94db5c6-b4nm5" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c") : secret "image-registry-tls" not found
May 28 00:11:55.463846 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.463825    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-certificates\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.464156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.464113    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a0045af-7f37-4ae7-89ec-7c66c799c21c-ca-trust-extracted\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.464156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.464131    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a0045af-7f37-4ae7-89ec-7c66c799c21c-trusted-ca\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.467596 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.467570    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a0045af-7f37-4ae7-89ec-7c66c799c21c-installation-pull-secrets\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.467685 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.467570    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a0045af-7f37-4ae7-89ec-7c66c799c21c-image-registry-private-configuration\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.475601 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.475574    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-bound-sa-token\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.475838 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.475822    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t46r\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-kube-api-access-5t46r\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.564385 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.564293    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2c7ab9-c05a-44d8-907a-377754c23b53-config-volume\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:11:55.564385 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.564348    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:11:55.564605 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:55.564451    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:11:55.564605 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.564525    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e2c7ab9-c05a-44d8-907a-377754c23b53-tmp-dir\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:11:55.564605 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:55.564538    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls podName:2e2c7ab9-c05a-44d8-907a-377754c23b53 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:56.064519379 +0000 UTC m=+35.685551386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls") pod "dns-default-4sq87" (UID: "2e2c7ab9-c05a-44d8-907a-377754c23b53") : secret "dns-default-metrics-tls" not found
May 28 00:11:55.564605 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.564566    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-46zj2\" (UniqueName: \"kubernetes.io/projected/2e2c7ab9-c05a-44d8-907a-377754c23b53-kube-api-access-46zj2\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:11:55.564816 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.564615    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert\") pod \"ingress-canary-tgwt7\" (UID: \"b1715227-4b2d-44dd-a153-51973245938a\") " pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:11:55.564816 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.564636    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8qqjb\" (UniqueName: \"kubernetes.io/projected/b1715227-4b2d-44dd-a153-51973245938a-kube-api-access-8qqjb\") pod \"ingress-canary-tgwt7\" (UID: \"b1715227-4b2d-44dd-a153-51973245938a\") " pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:11:55.564816 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:55.564748    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:11:55.564816 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:55.564815    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert podName:b1715227-4b2d-44dd-a153-51973245938a nodeName:}" failed. No retries permitted until 2026-05-28 00:11:56.064798288 +0000 UTC m=+35.685830288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert") pod "ingress-canary-tgwt7" (UID: "b1715227-4b2d-44dd-a153-51973245938a") : secret "canary-serving-cert" not found
May 28 00:11:55.564984 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.564872    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e2c7ab9-c05a-44d8-907a-377754c23b53-tmp-dir\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:11:55.564984 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.564958    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2c7ab9-c05a-44d8-907a-377754c23b53-config-volume\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:11:55.575623 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.575539    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46zj2\" (UniqueName: \"kubernetes.io/projected/2e2c7ab9-c05a-44d8-907a-377754c23b53-kube-api-access-46zj2\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:11:55.575763 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.575733    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qqjb\" (UniqueName: \"kubernetes.io/projected/b1715227-4b2d-44dd-a153-51973245938a-kube-api-access-8qqjb\") pod \"ingress-canary-tgwt7\" (UID: \"b1715227-4b2d-44dd-a153-51973245938a\") " pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:11:55.967004 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:55.966928    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:55.967482 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:55.967106    2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 00:11:55.967482 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:55.967127    2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79b94db5c6-b4nm5: secret "image-registry-tls" not found
May 28 00:11:55.967482 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:55.967204    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls podName:0a0045af-7f37-4ae7-89ec-7c66c799c21c nodeName:}" failed. No retries permitted until 2026-05-28 00:11:56.967183997 +0000 UTC m=+36.588215991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls") pod "image-registry-79b94db5c6-b4nm5" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c") : secret "image-registry-tls" not found
May 28 00:11:56.067313 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:56.067275    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert\") pod \"ingress-canary-tgwt7\" (UID: \"b1715227-4b2d-44dd-a153-51973245938a\") " pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:11:56.067496 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:56.067444    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:11:56.067496 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:56.067461    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:11:56.067649 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:56.067533    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert podName:b1715227-4b2d-44dd-a153-51973245938a nodeName:}" failed. No retries permitted until 2026-05-28 00:11:57.067499258 +0000 UTC m=+36.688531250 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert") pod "ingress-canary-tgwt7" (UID: "b1715227-4b2d-44dd-a153-51973245938a") : secret "canary-serving-cert" not found
May 28 00:11:56.067649 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:56.067589    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:11:56.067649 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:56.067630    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls podName:2e2c7ab9-c05a-44d8-907a-377754c23b53 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:57.06761833 +0000 UTC m=+36.688650329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls") pod "dns-default-4sq87" (UID: "2e2c7ab9-c05a-44d8-907a-377754c23b53") : secret "dns-default-metrics-tls" not found
May 28 00:11:56.975360 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:56.975322    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:56.975781 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:56.975487    2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 00:11:56.975781 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:56.975519    2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79b94db5c6-b4nm5: secret "image-registry-tls" not found
May 28 00:11:56.975781 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:56.975580    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls podName:0a0045af-7f37-4ae7-89ec-7c66c799c21c nodeName:}" failed. No retries permitted until 2026-05-28 00:11:58.975561793 +0000 UTC m=+38.596593787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls") pod "image-registry-79b94db5c6-b4nm5" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c") : secret "image-registry-tls" not found
May 28 00:11:57.012203 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:57.012170    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:11:57.012377 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:57.012170    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:11:57.012377 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:57.012175    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:11:57.015185 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:57.015163    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 28 00:11:57.016371 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:57.016350    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q24q2\""
May 28 00:11:57.016493 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:57.016373    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 28 00:11:57.016493 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:57.016377    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 28 00:11:57.016700 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:57.016681    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fp9tk\""
May 28 00:11:57.016700 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:57.016692    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 28 00:11:57.076456 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:57.076416    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:11:57.076658 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:57.076537    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert\") pod \"ingress-canary-tgwt7\" (UID: \"b1715227-4b2d-44dd-a153-51973245938a\") " pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:11:57.076658 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:57.076587    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:11:57.076777 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:57.076669    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:11:57.076777 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:57.076680    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls podName:2e2c7ab9-c05a-44d8-907a-377754c23b53 nodeName:}" failed. No retries permitted until 2026-05-28 00:11:59.076658514 +0000 UTC m=+38.697690507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls") pod "dns-default-4sq87" (UID: "2e2c7ab9-c05a-44d8-907a-377754c23b53") : secret "dns-default-metrics-tls" not found
May 28 00:11:57.076777 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:57.076731    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert podName:b1715227-4b2d-44dd-a153-51973245938a nodeName:}" failed. No retries permitted until 2026-05-28 00:11:59.076714464 +0000 UTC m=+38.697746457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert") pod "ingress-canary-tgwt7" (UID: "b1715227-4b2d-44dd-a153-51973245938a") : secret "canary-serving-cert" not found
May 28 00:11:58.993077 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:58.992828    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:11:58.993446 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:58.992974    2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 00:11:58.993446 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:58.993104    2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79b94db5c6-b4nm5: secret "image-registry-tls" not found
May 28 00:11:58.993446 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:58.993156    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls podName:0a0045af-7f37-4ae7-89ec-7c66c799c21c nodeName:}" failed. No retries permitted until 2026-05-28 00:12:02.993140992 +0000 UTC m=+42.614172993 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls") pod "image-registry-79b94db5c6-b4nm5" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c") : secret "image-registry-tls" not found
May 28 00:11:59.094023 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:59.093984    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert\") pod \"ingress-canary-tgwt7\" (UID: \"b1715227-4b2d-44dd-a153-51973245938a\") " pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:11:59.094188 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:59.094052    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:11:59.094188 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:59.094134    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:11:59.094188 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:59.094158    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:11:59.094304 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:59.094200    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert podName:b1715227-4b2d-44dd-a153-51973245938a nodeName:}" failed. No retries permitted until 2026-05-28 00:12:03.094183883 +0000 UTC m=+42.715215878 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert") pod "ingress-canary-tgwt7" (UID: "b1715227-4b2d-44dd-a153-51973245938a") : secret "canary-serving-cert" not found
May 28 00:11:59.094304 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:11:59.094216    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls podName:2e2c7ab9-c05a-44d8-907a-377754c23b53 nodeName:}" failed. No retries permitted until 2026-05-28 00:12:03.094209065 +0000 UTC m=+42.715241056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls") pod "dns-default-4sq87" (UID: "2e2c7ab9-c05a-44d8-907a-377754c23b53") : secret "dns-default-metrics-tls" not found
May 28 00:11:59.147998 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:59.147966    2576 generic.go:358] "Generic (PLEG): container finished" podID="d5d293d3-caa4-44cf-bae5-219a9e493213" containerID="f26263aecbcb37f43b8c7bc2f9adb67a42a78133b9f45a266cc20017fdf9a2b4" exitCode=0
May 28 00:11:59.148156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:11:59.148046    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9lmd" event={"ID":"d5d293d3-caa4-44cf-bae5-219a9e493213","Type":"ContainerDied","Data":"f26263aecbcb37f43b8c7bc2f9adb67a42a78133b9f45a266cc20017fdf9a2b4"}
May 28 00:12:00.000946 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:00.000909    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret\") pod \"global-pull-secret-syncer-h7v4h\" (UID: \"3a977914-7378-4daf-a7ca-b1e54548a760\") " pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:12:00.004443 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:00.004421    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a977914-7378-4daf-a7ca-b1e54548a760-original-pull-secret\") pod \"global-pull-secret-syncer-h7v4h\" (UID: \"3a977914-7378-4daf-a7ca-b1e54548a760\") " pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:12:00.033694 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:00.033663    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h7v4h"
May 28 00:12:00.154582 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:00.154046    2576 generic.go:358] "Generic (PLEG): container finished" podID="d5d293d3-caa4-44cf-bae5-219a9e493213" containerID="d75243b20b481b7a1f3633f9034ea3b91409b92b2531461d42483bce5edc1c2e" exitCode=0
May 28 00:12:00.154582 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:00.154344    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9lmd" event={"ID":"d5d293d3-caa4-44cf-bae5-219a9e493213","Type":"ContainerDied","Data":"d75243b20b481b7a1f3633f9034ea3b91409b92b2531461d42483bce5edc1c2e"}
May 28 00:12:00.204240 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:00.204204    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h7v4h"]
May 28 00:12:00.207828 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:12:00.207792    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a977914_7378_4daf_a7ca_b1e54548a760.slice/crio-1fdc1cf9716e732b80f4ab3a8b63838b5f43034a6c31d4e3d1fa94110cec9d5b WatchSource:0}: Error finding container 1fdc1cf9716e732b80f4ab3a8b63838b5f43034a6c31d4e3d1fa94110cec9d5b: Status 404 returned error can't find the container with id 1fdc1cf9716e732b80f4ab3a8b63838b5f43034a6c31d4e3d1fa94110cec9d5b
May 28 00:12:01.157042 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:01.157007    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h7v4h" event={"ID":"3a977914-7378-4daf-a7ca-b1e54548a760","Type":"ContainerStarted","Data":"1fdc1cf9716e732b80f4ab3a8b63838b5f43034a6c31d4e3d1fa94110cec9d5b"}
May 28 00:12:01.160175 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:01.160148    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r9lmd" event={"ID":"d5d293d3-caa4-44cf-bae5-219a9e493213","Type":"ContainerStarted","Data":"eda62b0d15ae80cb74d6a7f680ec4fc4d3ac49de0bd954df2dd8ae38311d485f"}
May 28 00:12:01.181360 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:01.181313    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r9lmd" podStartSLOduration=5.498077349 podStartE2EDuration="40.181299485s" podCreationTimestamp="2026-05-28 00:11:21 +0000 UTC" firstStartedPulling="2026-05-28 00:11:23.871660332 +0000 UTC m=+3.492692324" lastFinishedPulling="2026-05-28 00:11:58.554882465 +0000 UTC m=+38.175914460" observedRunningTime="2026-05-28 00:12:01.180251401 +0000 UTC m=+40.801283408" watchObservedRunningTime="2026-05-28 00:12:01.181299485 +0000 UTC m=+40.802331497"
May 28 00:12:03.024471 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:03.024436    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:12:03.024855 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:03.024599    2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 00:12:03.024855 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:03.024615    2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79b94db5c6-b4nm5: secret "image-registry-tls" not found
May 28 00:12:03.024855 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:03.024669    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls podName:0a0045af-7f37-4ae7-89ec-7c66c799c21c nodeName:}" failed. No retries permitted until 2026-05-28 00:12:11.024654646 +0000 UTC m=+50.645686637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls") pod "image-registry-79b94db5c6-b4nm5" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c") : secret "image-registry-tls" not found
May 28 00:12:03.125619 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:03.125577    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:12:03.125770 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:03.125680    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert\") pod \"ingress-canary-tgwt7\" (UID: \"b1715227-4b2d-44dd-a153-51973245938a\") " pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:12:03.125770 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:03.125751    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:12:03.125872 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:03.125823    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:12:03.125872 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:03.125828    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls podName:2e2c7ab9-c05a-44d8-907a-377754c23b53 nodeName:}" failed. No retries permitted until 2026-05-28 00:12:11.12580726 +0000 UTC m=+50.746839251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls") pod "dns-default-4sq87" (UID: "2e2c7ab9-c05a-44d8-907a-377754c23b53") : secret "dns-default-metrics-tls" not found
May 28 00:12:03.125957 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:03.125888    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert podName:b1715227-4b2d-44dd-a153-51973245938a nodeName:}" failed. No retries permitted until 2026-05-28 00:12:11.125874907 +0000 UTC m=+50.746906898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert") pod "ingress-canary-tgwt7" (UID: "b1715227-4b2d-44dd-a153-51973245938a") : secret "canary-serving-cert" not found
May 28 00:12:06.171705 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:06.171672    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h7v4h" event={"ID":"3a977914-7378-4daf-a7ca-b1e54548a760","Type":"ContainerStarted","Data":"83f6c25959872580b08030b83bdfae686714e9e0bacab244fcff31df2ab8cdb7"}
May 28 00:12:07.198404 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:07.196918    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h7v4h" podStartSLOduration=33.399459452 podStartE2EDuration="39.196893955s" podCreationTimestamp="2026-05-28 00:11:28 +0000 UTC" firstStartedPulling="2026-05-28 00:12:00.209726731 +0000 UTC m=+39.830758723" lastFinishedPulling="2026-05-28 00:12:06.007161232 +0000 UTC m=+45.628193226" observedRunningTime="2026-05-28 00:12:07.196087231 +0000 UTC m=+46.817119245" watchObservedRunningTime="2026-05-28 00:12:07.196893955 +0000 UTC m=+46.817925970"
May 28 00:12:11.080541 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:11.080481    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:12:11.080929 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:11.080626    2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 00:12:11.080929 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:11.080647    2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79b94db5c6-b4nm5: secret "image-registry-tls" not found
May 28 00:12:11.080929 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:11.080700    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls podName:0a0045af-7f37-4ae7-89ec-7c66c799c21c nodeName:}" failed. No retries permitted until 2026-05-28 00:12:27.080685271 +0000 UTC m=+66.701717263 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls") pod "image-registry-79b94db5c6-b4nm5" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c") : secret "image-registry-tls" not found
May 28 00:12:11.181366 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:11.181335    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert\") pod \"ingress-canary-tgwt7\" (UID: \"b1715227-4b2d-44dd-a153-51973245938a\") " pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:12:11.181524 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:11.181412    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:12:11.181524 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:11.181499    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:12:11.181639 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:11.181578    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:12:11.181639 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:11.181592    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert podName:b1715227-4b2d-44dd-a153-51973245938a nodeName:}" failed. No retries permitted until 2026-05-28 00:12:27.181570709 +0000 UTC m=+66.802602713 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert") pod "ingress-canary-tgwt7" (UID: "b1715227-4b2d-44dd-a153-51973245938a") : secret "canary-serving-cert" not found
May 28 00:12:11.181749 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:11.181640    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls podName:2e2c7ab9-c05a-44d8-907a-377754c23b53 nodeName:}" failed. No retries permitted until 2026-05-28 00:12:27.181620934 +0000 UTC m=+66.802652948 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls") pod "dns-default-4sq87" (UID: "2e2c7ab9-c05a-44d8-907a-377754c23b53") : secret "dns-default-metrics-tls" not found
May 28 00:12:21.147136 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:21.147105    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8gct7"
May 28 00:12:26.689280 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:26.689223    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs\") pod \"network-metrics-daemon-6b84s\" (UID: \"653f150e-1c4b-4dc5-a088-6ab0ad385350\") " pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:12:26.691943 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:26.691922    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 28 00:12:26.700107 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:26.700083    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 28 00:12:26.700257 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:26.700242    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs podName:653f150e-1c4b-4dc5-a088-6ab0ad385350 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:30.700219783 +0000 UTC m=+130.321251774 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs") pod "network-metrics-daemon-6b84s" (UID: "653f150e-1c4b-4dc5-a088-6ab0ad385350") : secret "metrics-daemon-secret" not found
May 28 00:12:26.790323 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:26.790290    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kt654\" (UniqueName: \"kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654\") pod \"network-check-target-xq6dj\" (UID: \"2e67c0ae-9c40-46d4-91a8-a5577a46bd2f\") " pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:12:26.793405 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:26.793384    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 28 00:12:26.803058 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:26.803038    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 28 00:12:26.814144 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:26.814122    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt654\" (UniqueName: \"kubernetes.io/projected/2e67c0ae-9c40-46d4-91a8-a5577a46bd2f-kube-api-access-kt654\") pod \"network-check-target-xq6dj\" (UID: \"2e67c0ae-9c40-46d4-91a8-a5577a46bd2f\") " pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:12:27.028691 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:27.028612    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q24q2\""
May 28 00:12:27.036007 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:27.035972    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:12:27.093694 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:27.093615    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:12:27.093845 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:27.093774    2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 00:12:27.093845 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:27.093792    2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79b94db5c6-b4nm5: secret "image-registry-tls" not found
May 28 00:12:27.093950 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:27.093888    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls podName:0a0045af-7f37-4ae7-89ec-7c66c799c21c nodeName:}" failed. No retries permitted until 2026-05-28 00:12:59.093870422 +0000 UTC m=+98.714902418 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls") pod "image-registry-79b94db5c6-b4nm5" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c") : secret "image-registry-tls" not found
May 28 00:12:27.151312 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:27.151281    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xq6dj"]
May 28 00:12:27.154357 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:12:27.154321    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e67c0ae_9c40_46d4_91a8_a5577a46bd2f.slice/crio-3fc761e9952ca043299909fdae1c687433d3643f4482188917ac849aca7d032c WatchSource:0}: Error finding container 3fc761e9952ca043299909fdae1c687433d3643f4482188917ac849aca7d032c: Status 404 returned error can't find the container with id 3fc761e9952ca043299909fdae1c687433d3643f4482188917ac849aca7d032c
May 28 00:12:27.194473 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:27.194439    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:12:27.194642 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:27.194523    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert\") pod \"ingress-canary-tgwt7\" (UID: \"b1715227-4b2d-44dd-a153-51973245938a\") " pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:12:27.194642 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:27.194595    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:12:27.194642 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:27.194624    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:12:27.194754 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:27.194662    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls podName:2e2c7ab9-c05a-44d8-907a-377754c23b53 nodeName:}" failed. No retries permitted until 2026-05-28 00:12:59.194645472 +0000 UTC m=+98.815677465 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls") pod "dns-default-4sq87" (UID: "2e2c7ab9-c05a-44d8-907a-377754c23b53") : secret "dns-default-metrics-tls" not found
May 28 00:12:27.194754 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:27.194678    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert podName:b1715227-4b2d-44dd-a153-51973245938a nodeName:}" failed. No retries permitted until 2026-05-28 00:12:59.194671511 +0000 UTC m=+98.815703503 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert") pod "ingress-canary-tgwt7" (UID: "b1715227-4b2d-44dd-a153-51973245938a") : secret "canary-serving-cert" not found
May 28 00:12:27.214435 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:27.214400    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xq6dj" event={"ID":"2e67c0ae-9c40-46d4-91a8-a5577a46bd2f","Type":"ContainerStarted","Data":"3fc761e9952ca043299909fdae1c687433d3643f4482188917ac849aca7d032c"}
May 28 00:12:31.223807 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:31.223770    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xq6dj" event={"ID":"2e67c0ae-9c40-46d4-91a8-a5577a46bd2f","Type":"ContainerStarted","Data":"b017ef2b8dcdc25ed6fed184101f092d1169b29572e1f9ccf8006035cb59082e"}
May 28 00:12:31.224179 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:31.223891    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:12:31.240522 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:31.240466    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xq6dj" podStartSLOduration=67.171000944 podStartE2EDuration="1m10.240452264s" podCreationTimestamp="2026-05-28 00:11:21 +0000 UTC" firstStartedPulling="2026-05-28 00:12:27.156532607 +0000 UTC m=+66.777564601" lastFinishedPulling="2026-05-28 00:12:30.22598393 +0000 UTC m=+69.847015921" observedRunningTime="2026-05-28 00:12:31.239990505 +0000 UTC m=+70.861022529" watchObservedRunningTime="2026-05-28 00:12:31.240452264 +0000 UTC m=+70.861484271"
May 28 00:12:59.121066 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:59.121020    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:12:59.121669 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:59.121197    2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 00:12:59.121669 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:59.121227    2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79b94db5c6-b4nm5: secret "image-registry-tls" not found
May 28 00:12:59.121669 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:59.121302    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls podName:0a0045af-7f37-4ae7-89ec-7c66c799c21c nodeName:}" failed. No retries permitted until 2026-05-28 00:14:03.121280878 +0000 UTC m=+162.742312870 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls") pod "image-registry-79b94db5c6-b4nm5" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c") : secret "image-registry-tls" not found
May 28 00:12:59.221978 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:59.221939    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:12:59.222147 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:12:59.222024    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert\") pod \"ingress-canary-tgwt7\" (UID: \"b1715227-4b2d-44dd-a153-51973245938a\") " pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:12:59.222147 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:59.222111    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 00:12:59.222220 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:59.222156    2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 00:12:59.222546 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:59.222525    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls podName:2e2c7ab9-c05a-44d8-907a-377754c23b53 nodeName:}" failed. No retries permitted until 2026-05-28 00:14:03.222477041 +0000 UTC m=+162.843509105 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls") pod "dns-default-4sq87" (UID: "2e2c7ab9-c05a-44d8-907a-377754c23b53") : secret "dns-default-metrics-tls" not found
May 28 00:12:59.222645 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:12:59.222577    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert podName:b1715227-4b2d-44dd-a153-51973245938a nodeName:}" failed. No retries permitted until 2026-05-28 00:14:03.222561415 +0000 UTC m=+162.843593407 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert") pod "ingress-canary-tgwt7" (UID: "b1715227-4b2d-44dd-a153-51973245938a") : secret "canary-serving-cert" not found
May 28 00:13:02.228151 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:02.228117    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xq6dj"
May 28 00:13:30.739678 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:30.739617    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs\") pod \"network-metrics-daemon-6b84s\" (UID: \"653f150e-1c4b-4dc5-a088-6ab0ad385350\") " pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:13:30.740189 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:30.739769    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 28 00:13:30.740189 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:30.739842    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs podName:653f150e-1c4b-4dc5-a088-6ab0ad385350 nodeName:}" failed. No retries permitted until 2026-05-28 00:15:32.739823438 +0000 UTC m=+252.360855429 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs") pod "network-metrics-daemon-6b84s" (UID: "653f150e-1c4b-4dc5-a088-6ab0ad385350") : secret "metrics-daemon-secret" not found
May 28 00:13:34.370396 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.370363    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb"]
May 28 00:13:34.373113 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.373094    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-24ksn"]
May 28 00:13:34.373256 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.373236    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb"
May 28 00:13:34.375842 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.375822    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-24ksn"
May 28 00:13:34.376225 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.376211    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 28 00:13:34.376426 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.376410    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 28 00:13:34.377273 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.377256    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 28 00:13:34.377347 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.377256    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2lhhz\""
May 28 00:13:34.377347 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.377290    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 28 00:13:34.378097 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.378083    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jtcrm\""
May 28 00:13:34.378486 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.378472    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 28 00:13:34.378597 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.378488    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 28 00:13:34.381165 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.381136    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-24ksn"]
May 28 00:13:34.384225 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.384205    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb"]
May 28 00:13:34.462519 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.462476    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"]
May 28 00:13:34.465486 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.465459    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"
May 28 00:13:34.467116 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.467095    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-nwc22"]
May 28 00:13:34.468731 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.468717    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 28 00:13:34.468956 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.468940    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 28 00:13:34.469012 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.468981    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-z5mnx\""
May 28 00:13:34.469060 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.469015    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 28 00:13:34.469633 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.469618    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-wfff8"]
May 28 00:13:34.469803 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.469788    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:34.472441 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.472422    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.473482 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.473461    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 28 00:13:34.473606 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.473484    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 28 00:13:34.474758 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.474733    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 28 00:13:34.474758 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.474741    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 28 00:13:34.474959 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.474944    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-df8tr\""
May 28 00:13:34.475422 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.475409    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
May 28 00:13:34.476210 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.476192    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-67hbz\""
May 28 00:13:34.477646 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.477625    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 28 00:13:34.478090 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.478072    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
May 28 00:13:34.478260 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.478243    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 28 00:13:34.479455 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.479303    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"]
May 28 00:13:34.481385 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.481359    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 28 00:13:34.488015 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.487969    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-nwc22"]
May 28 00:13:34.489165 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.489145    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-wfff8"]
May 28 00:13:34.490637 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.490613    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
May 28 00:13:34.564442 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls\") pod \"cluster-samples-operator-697587599-tr6nj\" (UID: \"0a851a7c-3e27-429d-a713-041330a90874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"
May 28 00:13:34.564657 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564449    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86b12396-57bb-4096-a674-699511beae9e-serving-cert\") pod \"console-operator-76dff48d59-nwc22\" (UID: \"86b12396-57bb-4096-a674-699511beae9e\") " pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:34.564657 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddl86\" (UniqueName: \"kubernetes.io/projected/43cdd4d1-8b9f-4b27-8280-5b668415be73-kube-api-access-ddl86\") pod \"kube-storage-version-migrator-operator-565644f49-j9fwb\" (UID: \"43cdd4d1-8b9f-4b27-8280-5b668415be73\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb"
May 28 00:13:34.564657 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564581    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86b12396-57bb-4096-a674-699511beae9e-trusted-ca\") pod \"console-operator-76dff48d59-nwc22\" (UID: \"86b12396-57bb-4096-a674-699511beae9e\") " pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:34.564657 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56g62\" (UniqueName: \"kubernetes.io/projected/354fa8c9-65f4-49f0-9793-12913a9b261d-kube-api-access-56g62\") pod \"volume-data-source-validator-5cf8c89d64-24ksn\" (UID: \"354fa8c9-65f4-49f0-9793-12913a9b261d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-24ksn"
May 28 00:13:34.564657 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm6p4\" (UniqueName: \"kubernetes.io/projected/0a851a7c-3e27-429d-a713-041330a90874-kube-api-access-zm6p4\") pod \"cluster-samples-operator-697587599-tr6nj\" (UID: \"0a851a7c-3e27-429d-a713-041330a90874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"
May 28 00:13:34.564859 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa182962-797b-4e21-a57f-7225c6bc806a-tmp\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.564859 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fa182962-797b-4e21-a57f-7225c6bc806a-snapshots\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.564859 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564705    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86b12396-57bb-4096-a674-699511beae9e-config\") pod \"console-operator-76dff48d59-nwc22\" (UID: \"86b12396-57bb-4096-a674-699511beae9e\") " pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:34.564859 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564773    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fa182962-797b-4e21-a57f-7225c6bc806a-serving-cert\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.564859 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk8rj\" (UniqueName: \"kubernetes.io/projected/86b12396-57bb-4096-a674-699511beae9e-kube-api-access-tk8rj\") pod \"console-operator-76dff48d59-nwc22\" (UID: \"86b12396-57bb-4096-a674-699511beae9e\") " pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:34.564859 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa182962-797b-4e21-a57f-7225c6bc806a-service-ca-bundle\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.564859 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa182962-797b-4e21-a57f-7225c6bc806a-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.565089 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43cdd4d1-8b9f-4b27-8280-5b668415be73-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-j9fwb\" (UID: \"43cdd4d1-8b9f-4b27-8280-5b668415be73\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb"
May 28 00:13:34.565089 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43cdd4d1-8b9f-4b27-8280-5b668415be73-config\") pod \"kube-storage-version-migrator-operator-565644f49-j9fwb\" (UID: \"43cdd4d1-8b9f-4b27-8280-5b668415be73\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb"
May 28 00:13:34.565089 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.564936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwl8k\" (UniqueName: \"kubernetes.io/projected/fa182962-797b-4e21-a57f-7225c6bc806a-kube-api-access-zwl8k\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.666240 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.666136    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43cdd4d1-8b9f-4b27-8280-5b668415be73-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-j9fwb\" (UID: \"43cdd4d1-8b9f-4b27-8280-5b668415be73\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb"
May 28 00:13:34.666240 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.666185    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43cdd4d1-8b9f-4b27-8280-5b668415be73-config\") pod \"kube-storage-version-migrator-operator-565644f49-j9fwb\" (UID: \"43cdd4d1-8b9f-4b27-8280-5b668415be73\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb"
May 28 00:13:34.666240 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.666212    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zwl8k\" (UniqueName: \"kubernetes.io/projected/fa182962-797b-4e21-a57f-7225c6bc806a-kube-api-access-zwl8k\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.666240 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.666239    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls\") pod \"cluster-samples-operator-697587599-tr6nj\" (UID: \"0a851a7c-3e27-429d-a713-041330a90874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"
May 28 00:13:34.666597 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.666267    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86b12396-57bb-4096-a674-699511beae9e-serving-cert\") pod \"console-operator-76dff48d59-nwc22\" (UID: \"86b12396-57bb-4096-a674-699511beae9e\") " pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:34.666597 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.666289    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ddl86\" (UniqueName: \"kubernetes.io/projected/43cdd4d1-8b9f-4b27-8280-5b668415be73-kube-api-access-ddl86\") pod \"kube-storage-version-migrator-operator-565644f49-j9fwb\" (UID: \"43cdd4d1-8b9f-4b27-8280-5b668415be73\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb"
May 28 00:13:34.666597 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.666320    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86b12396-57bb-4096-a674-699511beae9e-trusted-ca\") pod \"console-operator-76dff48d59-nwc22\" (UID: \"86b12396-57bb-4096-a674-699511beae9e\") " pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:34.666597 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:34.666362    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 00:13:34.666597 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:34.666451    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls podName:0a851a7c-3e27-429d-a713-041330a90874 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:35.166428843 +0000 UTC m=+134.787460851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls") pod "cluster-samples-operator-697587599-tr6nj" (UID: "0a851a7c-3e27-429d-a713-041330a90874") : secret "samples-operator-tls" not found
May 28 00:13:34.666597 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.666342    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56g62\" (UniqueName: \"kubernetes.io/projected/354fa8c9-65f4-49f0-9793-12913a9b261d-kube-api-access-56g62\") pod \"volume-data-source-validator-5cf8c89d64-24ksn\" (UID: \"354fa8c9-65f4-49f0-9793-12913a9b261d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-24ksn"
May 28 00:13:34.666980 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.666832    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/43cdd4d1-8b9f-4b27-8280-5b668415be73-config\") pod \"kube-storage-version-migrator-operator-565644f49-j9fwb\" (UID: \"43cdd4d1-8b9f-4b27-8280-5b668415be73\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb"
May 28 00:13:34.667085 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.667063    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zm6p4\" (UniqueName: \"kubernetes.io/projected/0a851a7c-3e27-429d-a713-041330a90874-kube-api-access-zm6p4\") pod \"cluster-samples-operator-697587599-tr6nj\" (UID: \"0a851a7c-3e27-429d-a713-041330a90874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"
May 28 00:13:34.667223 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.667192    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa182962-797b-4e21-a57f-7225c6bc806a-tmp\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.667354 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.667226    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fa182962-797b-4e21-a57f-7225c6bc806a-snapshots\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.667354 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.667266    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86b12396-57bb-4096-a674-699511beae9e-config\") pod \"console-operator-76dff48d59-nwc22\" (UID: \"86b12396-57bb-4096-a674-699511beae9e\") " pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:34.667604 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.667572    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86b12396-57bb-4096-a674-699511beae9e-trusted-ca\") pod \"console-operator-76dff48d59-nwc22\" (UID: \"86b12396-57bb-4096-a674-699511beae9e\") " pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:34.667759 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.667736    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fa182962-797b-4e21-a57f-7225c6bc806a-serving-cert\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.667843 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.667789    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tk8rj\" (UniqueName: \"kubernetes.io/projected/86b12396-57bb-4096-a674-699511beae9e-kube-api-access-tk8rj\") pod \"console-operator-76dff48d59-nwc22\" (UID: \"86b12396-57bb-4096-a674-699511beae9e\") " pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:34.667843 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.667822    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa182962-797b-4e21-a57f-7225c6bc806a-service-ca-bundle\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.667951 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.667864    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa182962-797b-4e21-a57f-7225c6bc806a-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.667951 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.667909    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86b12396-57bb-4096-a674-699511beae9e-config\") pod \"console-operator-76dff48d59-nwc22\" (UID: \"86b12396-57bb-4096-a674-699511beae9e\") " pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:34.668248 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.668202    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fa182962-797b-4e21-a57f-7225c6bc806a-snapshots\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.668248 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.668204    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa182962-797b-4e21-a57f-7225c6bc806a-tmp\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.668669 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.668642    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa182962-797b-4e21-a57f-7225c6bc806a-service-ca-bundle\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.668839 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.668818    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"]
May 28 00:13:34.668956 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.668935    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa182962-797b-4e21-a57f-7225c6bc806a-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.669141 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.669121    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86b12396-57bb-4096-a674-699511beae9e-serving-cert\") pod \"console-operator-76dff48d59-nwc22\" (UID: \"86b12396-57bb-4096-a674-699511beae9e\") " pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:34.669229 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.669182    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43cdd4d1-8b9f-4b27-8280-5b668415be73-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-j9fwb\" (UID: \"43cdd4d1-8b9f-4b27-8280-5b668415be73\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb"
May 28 00:13:34.670241 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.670212    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fa182962-797b-4e21-a57f-7225c6bc806a-serving-cert\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.673607 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.673590    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"
May 28 00:13:34.676213 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.676193    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bhf7n\""
May 28 00:13:34.676305 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.676215    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
May 28 00:13:34.676368 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.676303    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
May 28 00:13:34.678494 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.678475    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56g62\" (UniqueName: \"kubernetes.io/projected/354fa8c9-65f4-49f0-9793-12913a9b261d-kube-api-access-56g62\") pod \"volume-data-source-validator-5cf8c89d64-24ksn\" (UID: \"354fa8c9-65f4-49f0-9793-12913a9b261d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-24ksn"
May 28 00:13:34.682689 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.682665    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"]
May 28 00:13:34.685624 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.685602    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddl86\" (UniqueName: \"kubernetes.io/projected/43cdd4d1-8b9f-4b27-8280-5b668415be73-kube-api-access-ddl86\") pod \"kube-storage-version-migrator-operator-565644f49-j9fwb\" (UID: \"43cdd4d1-8b9f-4b27-8280-5b668415be73\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb"
May 28 00:13:34.685993 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.685977    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm6p4\" (UniqueName: \"kubernetes.io/projected/0a851a7c-3e27-429d-a713-041330a90874-kube-api-access-zm6p4\") pod \"cluster-samples-operator-697587599-tr6nj\" (UID: \"0a851a7c-3e27-429d-a713-041330a90874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"
May 28 00:13:34.686108 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.686086    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwl8k\" (UniqueName: \"kubernetes.io/projected/fa182962-797b-4e21-a57f-7225c6bc806a-kube-api-access-zwl8k\") pod \"insights-operator-78bdfb888c-wfff8\" (UID: \"fa182962-797b-4e21-a57f-7225c6bc806a\") " pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.687369 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.687349    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk8rj\" (UniqueName: \"kubernetes.io/projected/86b12396-57bb-4096-a674-699511beae9e-kube-api-access-tk8rj\") pod \"console-operator-76dff48d59-nwc22\" (UID: \"86b12396-57bb-4096-a674-699511beae9e\") " pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:34.690168 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.690148    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-24ksn"
May 28 00:13:34.769026 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.768995    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-bpdzw\" (UID: \"3c036ea7-0bb5-45d4-bacb-3b17d1156f0b\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"
May 28 00:13:34.769170 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.769088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bpdzw\" (UID: \"3c036ea7-0bb5-45d4-bacb-3b17d1156f0b\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"
May 28 00:13:34.784225 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.784194    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:34.793023 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.792994    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-wfff8"
May 28 00:13:34.809359 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.809276    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-24ksn"]
May 28 00:13:34.812545 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:13:34.812475    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354fa8c9_65f4_49f0_9793_12913a9b261d.slice/crio-19259c461858d84d6c7ebdddcffcac8662786ae31340267b676828c7f5b9d230 WatchSource:0}: Error finding container 19259c461858d84d6c7ebdddcffcac8662786ae31340267b676828c7f5b9d230: Status 404 returned error can't find the container with id 19259c461858d84d6c7ebdddcffcac8662786ae31340267b676828c7f5b9d230
May 28 00:13:34.869568 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.869535    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bpdzw\" (UID: \"3c036ea7-0bb5-45d4-bacb-3b17d1156f0b\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"
May 28 00:13:34.869727 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.869656    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-bpdzw\" (UID: \"3c036ea7-0bb5-45d4-bacb-3b17d1156f0b\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"
May 28 00:13:34.869727 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:34.869682    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 28 00:13:34.869836 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:34.869753    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert podName:3c036ea7-0bb5-45d4-bacb-3b17d1156f0b nodeName:}" failed. No retries permitted until 2026-05-28 00:13:35.369734492 +0000 UTC m=+134.990766487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-bpdzw" (UID: "3c036ea7-0bb5-45d4-bacb-3b17d1156f0b") : secret "networking-console-plugin-cert" not found
May 28 00:13:34.870326 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.870305    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-bpdzw\" (UID: \"3c036ea7-0bb5-45d4-bacb-3b17d1156f0b\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"
May 28 00:13:34.919955 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.919912    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-nwc22"]
May 28 00:13:34.922252 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:13:34.922217    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b12396_57bb_4096_a674_699511beae9e.slice/crio-0cd2335a55b0d574e509f419e0bd9953aa1a25918deaf5e74b73995e161e782e WatchSource:0}: Error finding container 0cd2335a55b0d574e509f419e0bd9953aa1a25918deaf5e74b73995e161e782e: Status 404 returned error can't find the container with id 0cd2335a55b0d574e509f419e0bd9953aa1a25918deaf5e74b73995e161e782e
May 28 00:13:34.933784 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.933761    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-wfff8"]
May 28 00:13:34.936140 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:13:34.936116    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa182962_797b_4e21_a57f_7225c6bc806a.slice/crio-1ac08394dddc6b81749ff1c6e89e8c90d5e21b065125cade2f44061b13b6305e WatchSource:0}: Error finding container 1ac08394dddc6b81749ff1c6e89e8c90d5e21b065125cade2f44061b13b6305e: Status 404 returned error can't find the container with id 1ac08394dddc6b81749ff1c6e89e8c90d5e21b065125cade2f44061b13b6305e
May 28 00:13:34.984417 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:34.984383    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb"
May 28 00:13:35.103451 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:35.103378    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb"]
May 28 00:13:35.105757 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:13:35.105734    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cdd4d1_8b9f_4b27_8280_5b668415be73.slice/crio-8936d468c2f5f79c6af4dac18eb5b46db6ee673ddef290123103a8b69c137977 WatchSource:0}: Error finding container 8936d468c2f5f79c6af4dac18eb5b46db6ee673ddef290123103a8b69c137977: Status 404 returned error can't find the container with id 8936d468c2f5f79c6af4dac18eb5b46db6ee673ddef290123103a8b69c137977
May 28 00:13:35.172107 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:35.172002    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls\") pod \"cluster-samples-operator-697587599-tr6nj\" (UID: \"0a851a7c-3e27-429d-a713-041330a90874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"
May 28 00:13:35.172260 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:35.172161    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 00:13:35.172260 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:35.172231    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls podName:0a851a7c-3e27-429d-a713-041330a90874 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:36.172212571 +0000 UTC m=+135.793244561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls") pod "cluster-samples-operator-697587599-tr6nj" (UID: "0a851a7c-3e27-429d-a713-041330a90874") : secret "samples-operator-tls" not found
May 28 00:13:35.347497 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:35.347463    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-nwc22" event={"ID":"86b12396-57bb-4096-a674-699511beae9e","Type":"ContainerStarted","Data":"0cd2335a55b0d574e509f419e0bd9953aa1a25918deaf5e74b73995e161e782e"}
May 28 00:13:35.348427 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:35.348401    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-24ksn" event={"ID":"354fa8c9-65f4-49f0-9793-12913a9b261d","Type":"ContainerStarted","Data":"19259c461858d84d6c7ebdddcffcac8662786ae31340267b676828c7f5b9d230"}
May 28 00:13:35.349306 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:35.349286    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-wfff8" event={"ID":"fa182962-797b-4e21-a57f-7225c6bc806a","Type":"ContainerStarted","Data":"1ac08394dddc6b81749ff1c6e89e8c90d5e21b065125cade2f44061b13b6305e"}
May 28 00:13:35.350169 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:35.350151    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb" event={"ID":"43cdd4d1-8b9f-4b27-8280-5b668415be73","Type":"ContainerStarted","Data":"8936d468c2f5f79c6af4dac18eb5b46db6ee673ddef290123103a8b69c137977"}
May 28 00:13:35.373777 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:35.373742    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bpdzw\" (UID: \"3c036ea7-0bb5-45d4-bacb-3b17d1156f0b\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"
May 28 00:13:35.374210 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:35.373904    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 28 00:13:35.374210 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:35.373966    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert podName:3c036ea7-0bb5-45d4-bacb-3b17d1156f0b nodeName:}" failed. No retries permitted until 2026-05-28 00:13:36.373948323 +0000 UTC m=+135.994980320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-bpdzw" (UID: "3c036ea7-0bb5-45d4-bacb-3b17d1156f0b") : secret "networking-console-plugin-cert" not found
May 28 00:13:36.180564 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:36.180520    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls\") pod \"cluster-samples-operator-697587599-tr6nj\" (UID: \"0a851a7c-3e27-429d-a713-041330a90874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"
May 28 00:13:36.180761 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:36.180646    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 00:13:36.180761 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:36.180727    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls podName:0a851a7c-3e27-429d-a713-041330a90874 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:38.180703245 +0000 UTC m=+137.801735241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls") pod "cluster-samples-operator-697587599-tr6nj" (UID: "0a851a7c-3e27-429d-a713-041330a90874") : secret "samples-operator-tls" not found
May 28 00:13:36.383078 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:36.383039    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bpdzw\" (UID: \"3c036ea7-0bb5-45d4-bacb-3b17d1156f0b\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"
May 28 00:13:36.383581 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:36.383220    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 28 00:13:36.383581 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:36.383303    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert podName:3c036ea7-0bb5-45d4-bacb-3b17d1156f0b nodeName:}" failed. No retries permitted until 2026-05-28 00:13:38.383281218 +0000 UTC m=+138.004313214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-bpdzw" (UID: "3c036ea7-0bb5-45d4-bacb-3b17d1156f0b") : secret "networking-console-plugin-cert" not found
May 28 00:13:37.355216 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:37.355177    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-24ksn" event={"ID":"354fa8c9-65f4-49f0-9793-12913a9b261d","Type":"ContainerStarted","Data":"cf06e12449fae8643edc9efbe08feec9ba347d9c3414d77526662e7adb90e925"}
May 28 00:13:37.371585 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:37.371538    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-24ksn" podStartSLOduration=1.893242929 podStartE2EDuration="3.371519658s" podCreationTimestamp="2026-05-28 00:13:34 +0000 UTC" firstStartedPulling="2026-05-28 00:13:34.814529107 +0000 UTC m=+134.435561111" lastFinishedPulling="2026-05-28 00:13:36.292805839 +0000 UTC m=+135.913837840" observedRunningTime="2026-05-28 00:13:37.370796671 +0000 UTC m=+136.991828717" watchObservedRunningTime="2026-05-28 00:13:37.371519658 +0000 UTC m=+136.992551666"
May 28 00:13:38.198323 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:38.198273    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls\") pod \"cluster-samples-operator-697587599-tr6nj\" (UID: \"0a851a7c-3e27-429d-a713-041330a90874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"
May 28 00:13:38.198785 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:38.198435    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 00:13:38.198785 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:38.198542    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls podName:0a851a7c-3e27-429d-a713-041330a90874 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:42.198523334 +0000 UTC m=+141.819555329 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls") pod "cluster-samples-operator-697587599-tr6nj" (UID: "0a851a7c-3e27-429d-a713-041330a90874") : secret "samples-operator-tls" not found
May 28 00:13:38.399106 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:38.399059    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bpdzw\" (UID: \"3c036ea7-0bb5-45d4-bacb-3b17d1156f0b\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"
May 28 00:13:38.399294 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:38.399240    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 28 00:13:38.399369 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:38.399335    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert podName:3c036ea7-0bb5-45d4-bacb-3b17d1156f0b nodeName:}" failed. No retries permitted until 2026-05-28 00:13:42.399311997 +0000 UTC m=+142.020344005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-bpdzw" (UID: "3c036ea7-0bb5-45d4-bacb-3b17d1156f0b") : secret "networking-console-plugin-cert" not found
May 28 00:13:39.361847 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:39.361817    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/0.log"
May 28 00:13:39.362306 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:39.361866    2576 generic.go:358] "Generic (PLEG): container finished" podID="86b12396-57bb-4096-a674-699511beae9e" containerID="6e8cd8488b48ffddaaaebfae5e539874d953e0358ff84995a2d7a10efdb2907f" exitCode=255
May 28 00:13:39.362306 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:39.361959    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-nwc22" event={"ID":"86b12396-57bb-4096-a674-699511beae9e","Type":"ContainerDied","Data":"6e8cd8488b48ffddaaaebfae5e539874d953e0358ff84995a2d7a10efdb2907f"}
May 28 00:13:39.362306 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:39.362181    2576 scope.go:117] "RemoveContainer" containerID="6e8cd8488b48ffddaaaebfae5e539874d953e0358ff84995a2d7a10efdb2907f"
May 28 00:13:39.363374 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:39.363352    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-wfff8" event={"ID":"fa182962-797b-4e21-a57f-7225c6bc806a","Type":"ContainerStarted","Data":"e2b51efbd1367a1c36ccc64969fe837d8d1c29fe1a95b7dccbe8352b6ce2d1dc"}
May 28 00:13:39.367454 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:39.367425    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb" event={"ID":"43cdd4d1-8b9f-4b27-8280-5b668415be73","Type":"ContainerStarted","Data":"318b68c60e3919c0a25ea5c581fa52e3e9a3f6af066601cfae781f3c7e26457a"}
May 28 00:13:39.395238 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:39.395183    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-wfff8" podStartSLOduration=1.7013581960000002 podStartE2EDuration="5.3951687s" podCreationTimestamp="2026-05-28 00:13:34 +0000 UTC" firstStartedPulling="2026-05-28 00:13:34.93800172 +0000 UTC m=+134.559033712" lastFinishedPulling="2026-05-28 00:13:38.631812216 +0000 UTC m=+138.252844216" observedRunningTime="2026-05-28 00:13:39.394473126 +0000 UTC m=+139.015505163" watchObservedRunningTime="2026-05-28 00:13:39.3951687 +0000 UTC m=+139.016200740"
May 28 00:13:39.412564 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:39.412412    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb" podStartSLOduration=1.88998359 podStartE2EDuration="5.412394501s" podCreationTimestamp="2026-05-28 00:13:34 +0000 UTC" firstStartedPulling="2026-05-28 00:13:35.107481401 +0000 UTC m=+134.728513393" lastFinishedPulling="2026-05-28 00:13:38.62989231 +0000 UTC m=+138.250924304" observedRunningTime="2026-05-28 00:13:39.412216031 +0000 UTC m=+139.033248045" watchObservedRunningTime="2026-05-28 00:13:39.412394501 +0000 UTC m=+139.033426516"
May 28 00:13:40.371798 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:40.371771    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:13:40.372225 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:40.372147    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/0.log"
May 28 00:13:40.372225 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:40.372189    2576 generic.go:358] "Generic (PLEG): container finished" podID="86b12396-57bb-4096-a674-699511beae9e" containerID="33eacddbc9e562f3be7df25807e4b55866f27d75d73f0399647de0d5db911f1f" exitCode=255
May 28 00:13:40.372303 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:40.372264    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-nwc22" event={"ID":"86b12396-57bb-4096-a674-699511beae9e","Type":"ContainerDied","Data":"33eacddbc9e562f3be7df25807e4b55866f27d75d73f0399647de0d5db911f1f"}
May 28 00:13:40.372343 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:40.372313    2576 scope.go:117] "RemoveContainer" containerID="6e8cd8488b48ffddaaaebfae5e539874d953e0358ff84995a2d7a10efdb2907f"
May 28 00:13:40.372558 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:40.372539    2576 scope.go:117] "RemoveContainer" containerID="33eacddbc9e562f3be7df25807e4b55866f27d75d73f0399647de0d5db911f1f"
May 28 00:13:40.372772 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:40.372752    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-nwc22_openshift-console-operator(86b12396-57bb-4096-a674-699511beae9e)\"" pod="openshift-console-operator/console-operator-76dff48d59-nwc22" podUID="86b12396-57bb-4096-a674-699511beae9e"
May 28 00:13:41.291085 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.291036    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-w8pmx"]
May 28 00:13:41.295362 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.295337    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-w8pmx"
May 28 00:13:41.297838 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.297809    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
May 28 00:13:41.298948 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.298930    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
May 28 00:13:41.299046 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.299012    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-stpk7\""
May 28 00:13:41.301204 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.301174    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-w8pmx"]
May 28 00:13:41.324541 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.324492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgt95\" (UniqueName: \"kubernetes.io/projected/e29b5003-2cae-4189-8a90-9508ac3a7570-kube-api-access-rgt95\") pod \"migrator-b985f5d89-w8pmx\" (UID: \"e29b5003-2cae-4189-8a90-9508ac3a7570\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-w8pmx"
May 28 00:13:41.376337 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.376311    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:13:41.376721 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.376676    2576 scope.go:117] "RemoveContainer" containerID="33eacddbc9e562f3be7df25807e4b55866f27d75d73f0399647de0d5db911f1f"
May 28 00:13:41.376849 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:41.376831    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-nwc22_openshift-console-operator(86b12396-57bb-4096-a674-699511beae9e)\"" pod="openshift-console-operator/console-operator-76dff48d59-nwc22" podUID="86b12396-57bb-4096-a674-699511beae9e"
May 28 00:13:41.425970 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.425924    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rgt95\" (UniqueName: \"kubernetes.io/projected/e29b5003-2cae-4189-8a90-9508ac3a7570-kube-api-access-rgt95\") pod \"migrator-b985f5d89-w8pmx\" (UID: \"e29b5003-2cae-4189-8a90-9508ac3a7570\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-w8pmx"
May 28 00:13:41.434076 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.434044    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgt95\" (UniqueName: \"kubernetes.io/projected/e29b5003-2cae-4189-8a90-9508ac3a7570-kube-api-access-rgt95\") pod \"migrator-b985f5d89-w8pmx\" (UID: \"e29b5003-2cae-4189-8a90-9508ac3a7570\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-w8pmx"
May 28 00:13:41.492485 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.492452    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-9c7dr"]
May 28 00:13:41.495443 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.495426    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-9c7dr"
May 28 00:13:41.498125 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.498096    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 28 00:13:41.498125 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.498096    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 28 00:13:41.498307 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.498239    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 28 00:13:41.498380 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.498366    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ztx7d\""
May 28 00:13:41.498515 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.498491    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 28 00:13:41.502530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.502495    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-9c7dr"]
May 28 00:13:41.526969 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.526932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e299ef6a-1c2f-4ecd-8043-0f91566212b9-signing-key\") pod \"service-ca-67d667c7d-9c7dr\" (UID: \"e299ef6a-1c2f-4ecd-8043-0f91566212b9\") " pod="openshift-service-ca/service-ca-67d667c7d-9c7dr"
May 28 00:13:41.527136 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.527002    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e299ef6a-1c2f-4ecd-8043-0f91566212b9-signing-cabundle\") pod \"service-ca-67d667c7d-9c7dr\" (UID: \"e299ef6a-1c2f-4ecd-8043-0f91566212b9\") " pod="openshift-service-ca/service-ca-67d667c7d-9c7dr"
May 28 00:13:41.527136 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.527023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8lzs\" (UniqueName: \"kubernetes.io/projected/e299ef6a-1c2f-4ecd-8043-0f91566212b9-kube-api-access-k8lzs\") pod \"service-ca-67d667c7d-9c7dr\" (UID: \"e299ef6a-1c2f-4ecd-8043-0f91566212b9\") " pod="openshift-service-ca/service-ca-67d667c7d-9c7dr"
May 28 00:13:41.605196 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.605114    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-w8pmx"
May 28 00:13:41.628373 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.628332    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e299ef6a-1c2f-4ecd-8043-0f91566212b9-signing-key\") pod \"service-ca-67d667c7d-9c7dr\" (UID: \"e299ef6a-1c2f-4ecd-8043-0f91566212b9\") " pod="openshift-service-ca/service-ca-67d667c7d-9c7dr"
May 28 00:13:41.628553 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.628449    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e299ef6a-1c2f-4ecd-8043-0f91566212b9-signing-cabundle\") pod \"service-ca-67d667c7d-9c7dr\" (UID: \"e299ef6a-1c2f-4ecd-8043-0f91566212b9\") " pod="openshift-service-ca/service-ca-67d667c7d-9c7dr"
May 28 00:13:41.628553 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.628479    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k8lzs\" (UniqueName: \"kubernetes.io/projected/e299ef6a-1c2f-4ecd-8043-0f91566212b9-kube-api-access-k8lzs\") pod \"service-ca-67d667c7d-9c7dr\" (UID: \"e299ef6a-1c2f-4ecd-8043-0f91566212b9\") " pod="openshift-service-ca/service-ca-67d667c7d-9c7dr"
May 28 00:13:41.629105 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.629083    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e299ef6a-1c2f-4ecd-8043-0f91566212b9-signing-cabundle\") pod \"service-ca-67d667c7d-9c7dr\" (UID: \"e299ef6a-1c2f-4ecd-8043-0f91566212b9\") " pod="openshift-service-ca/service-ca-67d667c7d-9c7dr"
May 28 00:13:41.630717 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.630693    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e299ef6a-1c2f-4ecd-8043-0f91566212b9-signing-key\") pod \"service-ca-67d667c7d-9c7dr\" (UID: \"e299ef6a-1c2f-4ecd-8043-0f91566212b9\") " pod="openshift-service-ca/service-ca-67d667c7d-9c7dr"
May 28 00:13:41.637767 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.637742    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8lzs\" (UniqueName: \"kubernetes.io/projected/e299ef6a-1c2f-4ecd-8043-0f91566212b9-kube-api-access-k8lzs\") pod \"service-ca-67d667c7d-9c7dr\" (UID: \"e299ef6a-1c2f-4ecd-8043-0f91566212b9\") " pod="openshift-service-ca/service-ca-67d667c7d-9c7dr"
May 28 00:13:41.722699 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.722669    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-w8pmx"]
May 28 00:13:41.725547 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:13:41.725522    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29b5003_2cae_4189_8a90_9508ac3a7570.slice/crio-2b6485797822d5afe0a0e533ca4b331e987fea7118118384cc27e6e7683ecbc7 WatchSource:0}: Error finding container 2b6485797822d5afe0a0e533ca4b331e987fea7118118384cc27e6e7683ecbc7: Status 404 returned error can't find the container with id 2b6485797822d5afe0a0e533ca4b331e987fea7118118384cc27e6e7683ecbc7
May 28 00:13:41.805410 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.805381    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-9c7dr"
May 28 00:13:41.922159 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:41.922138    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-9c7dr"]
May 28 00:13:41.923895 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:13:41.923872    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode299ef6a_1c2f_4ecd_8043_0f91566212b9.slice/crio-9947aefd08d16f8fb109682cb398e8ee29d5b3bdf3318bc54f6e02ec50f15ef9 WatchSource:0}: Error finding container 9947aefd08d16f8fb109682cb398e8ee29d5b3bdf3318bc54f6e02ec50f15ef9: Status 404 returned error can't find the container with id 9947aefd08d16f8fb109682cb398e8ee29d5b3bdf3318bc54f6e02ec50f15ef9
May 28 00:13:42.234463 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:42.234376    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls\") pod \"cluster-samples-operator-697587599-tr6nj\" (UID: \"0a851a7c-3e27-429d-a713-041330a90874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"
May 28 00:13:42.234615 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:42.234532    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 00:13:42.234615 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:42.234595    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls podName:0a851a7c-3e27-429d-a713-041330a90874 nodeName:}" failed. No retries permitted until 2026-05-28 00:13:50.234579037 +0000 UTC m=+149.855611028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls") pod "cluster-samples-operator-697587599-tr6nj" (UID: "0a851a7c-3e27-429d-a713-041330a90874") : secret "samples-operator-tls" not found
May 28 00:13:42.383460 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:42.383421    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-9c7dr" event={"ID":"e299ef6a-1c2f-4ecd-8043-0f91566212b9","Type":"ContainerStarted","Data":"9947aefd08d16f8fb109682cb398e8ee29d5b3bdf3318bc54f6e02ec50f15ef9"}
May 28 00:13:42.384306 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:42.384286    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-w8pmx" event={"ID":"e29b5003-2cae-4189-8a90-9508ac3a7570","Type":"ContainerStarted","Data":"2b6485797822d5afe0a0e533ca4b331e987fea7118118384cc27e6e7683ecbc7"}
May 28 00:13:42.436264 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:42.436221    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bpdzw\" (UID: \"3c036ea7-0bb5-45d4-bacb-3b17d1156f0b\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"
May 28 00:13:42.436405 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:42.436377    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 28 00:13:42.436455 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:42.436444    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert podName:3c036ea7-0bb5-45d4-bacb-3b17d1156f0b nodeName:}" failed. No retries permitted until 2026-05-28 00:13:50.436428029 +0000 UTC m=+150.057460020 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-bpdzw" (UID: "3c036ea7-0bb5-45d4-bacb-3b17d1156f0b") : secret "networking-console-plugin-cert" not found
May 28 00:13:42.795975 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:42.795944    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tmc88_d74ab08c-2275-4054-9365-ba62ea412c17/dns-node-resolver/0.log"
May 28 00:13:43.388526 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:43.388425    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-w8pmx" event={"ID":"e29b5003-2cae-4189-8a90-9508ac3a7570","Type":"ContainerStarted","Data":"3e0d16e96c66ffae7d91866ce34e825daf5dce1b6582c648bf3ee67ff9ed248e"}
May 28 00:13:43.388526 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:43.388470    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-w8pmx" event={"ID":"e29b5003-2cae-4189-8a90-9508ac3a7570","Type":"ContainerStarted","Data":"c7b52c22c961968e00e907f650141e1d2a521101e85b7083c74bbd9ad8df951a"}
May 28 00:13:43.406942 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:43.406890    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-w8pmx" podStartSLOduration=1.032797266 podStartE2EDuration="2.406875166s" podCreationTimestamp="2026-05-28 00:13:41 +0000 UTC" firstStartedPulling="2026-05-28 00:13:41.727305311 +0000 UTC m=+141.348337305" lastFinishedPulling="2026-05-28 00:13:43.10138321 +0000 UTC m=+142.722415205" observedRunningTime="2026-05-28 00:13:43.405982297 +0000 UTC m=+143.027014309" watchObservedRunningTime="2026-05-28 00:13:43.406875166 +0000 UTC m=+143.027907178"
May 28 00:13:43.593435 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:43.593406    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mvfbv_a2f133ad-ace2-417a-a450-7f917265d112/node-ca/0.log"
May 28 00:13:44.784364 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:44.784325    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:44.784742 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:44.784384    2576 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:44.784894 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:44.784878    2576 scope.go:117] "RemoveContainer" containerID="33eacddbc9e562f3be7df25807e4b55866f27d75d73f0399647de0d5db911f1f"
May 28 00:13:44.785106 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:44.785085    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-nwc22_openshift-console-operator(86b12396-57bb-4096-a674-699511beae9e)\"" pod="openshift-console-operator/console-operator-76dff48d59-nwc22" podUID="86b12396-57bb-4096-a674-699511beae9e"
May 28 00:13:45.395040 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:45.394957    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-9c7dr" event={"ID":"e299ef6a-1c2f-4ecd-8043-0f91566212b9","Type":"ContainerStarted","Data":"f279d70c33b413996aa5c56cfef9cc14b5ea6ddb1e1f2252a9aa96b6645db9c5"}
May 28 00:13:45.410359 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:45.410305    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-9c7dr" podStartSLOduration=1.876360339 podStartE2EDuration="4.410290019s" podCreationTimestamp="2026-05-28 00:13:41 +0000 UTC" firstStartedPulling="2026-05-28 00:13:41.92565553 +0000 UTC m=+141.546687521" lastFinishedPulling="2026-05-28 00:13:44.459585198 +0000 UTC m=+144.080617201" observedRunningTime="2026-05-28 00:13:45.409988726 +0000 UTC m=+145.031020751" watchObservedRunningTime="2026-05-28 00:13:45.410290019 +0000 UTC m=+145.031322012"
May 28 00:13:50.304676 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:50.304626    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls\") pod \"cluster-samples-operator-697587599-tr6nj\" (UID: \"0a851a7c-3e27-429d-a713-041330a90874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"
May 28 00:13:50.307206 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:50.307166    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0a851a7c-3e27-429d-a713-041330a90874-samples-operator-tls\") pod \"cluster-samples-operator-697587599-tr6nj\" (UID: \"0a851a7c-3e27-429d-a713-041330a90874\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"
May 28 00:13:50.376645 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:50.376605    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"
May 28 00:13:50.495869 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:50.495839    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj"]
May 28 00:13:50.507811 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:50.507127    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bpdzw\" (UID: \"3c036ea7-0bb5-45d4-bacb-3b17d1156f0b\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"
May 28 00:13:50.507811 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:50.507359    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 28 00:13:50.507811 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:50.507425    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert podName:3c036ea7-0bb5-45d4-bacb-3b17d1156f0b nodeName:}" failed. No retries permitted until 2026-05-28 00:14:06.507406806 +0000 UTC m=+166.128438803 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-bpdzw" (UID: "3c036ea7-0bb5-45d4-bacb-3b17d1156f0b") : secret "networking-console-plugin-cert" not found
May 28 00:13:51.412144 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:51.412110    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj" event={"ID":"0a851a7c-3e27-429d-a713-041330a90874","Type":"ContainerStarted","Data":"028dec86a34ee2ff87444835d1e2977fca9cc8756b60233c779b5a09d3a81bde"}
May 28 00:13:52.420759 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:52.420689    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj" event={"ID":"0a851a7c-3e27-429d-a713-041330a90874","Type":"ContainerStarted","Data":"ae3394c8573d1e50a484e2cb6d332cfe104fd4654f754de3dfe0830f2d40528d"}
May 28 00:13:52.420759 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:52.420734    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj" event={"ID":"0a851a7c-3e27-429d-a713-041330a90874","Type":"ContainerStarted","Data":"a1cbd5c1619a2fa277761698378828759b95814d5d691eb7722874c39967e239"}
May 28 00:13:52.438397 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:52.438353    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-tr6nj" podStartSLOduration=16.718789408 podStartE2EDuration="18.438339647s" podCreationTimestamp="2026-05-28 00:13:34 +0000 UTC" firstStartedPulling="2026-05-28 00:13:50.554012362 +0000 UTC m=+150.175044352" lastFinishedPulling="2026-05-28 00:13:52.273562597 +0000 UTC m=+151.894594591" observedRunningTime="2026-05-28 00:13:52.437598453 +0000 UTC m=+152.058630467" watchObservedRunningTime="2026-05-28 00:13:52.438339647 +0000 UTC m=+152.059371671"
May 28 00:13:56.012388 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:56.012358    2576 scope.go:117] "RemoveContainer" containerID="33eacddbc9e562f3be7df25807e4b55866f27d75d73f0399647de0d5db911f1f"
May 28 00:13:56.432245 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:56.432216    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:13:56.432437 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:56.432287    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-nwc22" event={"ID":"86b12396-57bb-4096-a674-699511beae9e","Type":"ContainerStarted","Data":"3f693f72fd1748797a072565fbdbf4999ff6fa85afc61d511f3c8b6315259bbe"}
May 28 00:13:56.432570 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:56.432553    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:56.457947 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:56.457894    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-nwc22" podStartSLOduration=18.756614646 podStartE2EDuration="22.457878295s" podCreationTimestamp="2026-05-28 00:13:34 +0000 UTC" firstStartedPulling="2026-05-28 00:13:34.924180287 +0000 UTC m=+134.545212278" lastFinishedPulling="2026-05-28 00:13:38.625443907 +0000 UTC m=+138.246475927" observedRunningTime="2026-05-28 00:13:56.457368272 +0000 UTC m=+156.078400286" watchObservedRunningTime="2026-05-28 00:13:56.457878295 +0000 UTC m=+156.078910308"
May 28 00:13:57.309629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:57.309600    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-nwc22"
May 28 00:13:58.269386 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:58.269341    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5" podUID="0a0045af-7f37-4ae7-89ec-7c66c799c21c"
May 28 00:13:58.286594 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:58.286558    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4sq87" podUID="2e2c7ab9-c05a-44d8-907a-377754c23b53"
May 28 00:13:58.292727 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:13:58.292702    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-tgwt7" podUID="b1715227-4b2d-44dd-a153-51973245938a"
May 28 00:13:58.436979 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:58.436943    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:13:58.437402 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:58.436989    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:13:58.437402 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:13:58.437020    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4sq87"
May 28 00:14:00.040914 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:14:00.040846    2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6b84s" podUID="653f150e-1c4b-4dc5-a088-6ab0ad385350"
May 28 00:14:01.750533 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:01.750474    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-62sqq"]
May 28 00:14:01.753909 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:01.753884    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:01.756522 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:01.756485    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 28 00:14:01.756644 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:01.756523    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 28 00:14:01.757628 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:01.757609    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rdsbx\""
May 28 00:14:01.769197 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:01.769171    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-62sqq"]
May 28 00:14:01.903304 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:01.903265    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e28953c-9a22-4850-941c-e9aabd07aef4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-62sqq\" (UID: \"6e28953c-9a22-4850-941c-e9aabd07aef4\") " pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:01.903304 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:01.903305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e28953c-9a22-4850-941c-e9aabd07aef4-crio-socket\") pod \"insights-runtime-extractor-62sqq\" (UID: \"6e28953c-9a22-4850-941c-e9aabd07aef4\") " pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:01.903527 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:01.903411    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e28953c-9a22-4850-941c-e9aabd07aef4-data-volume\") pod \"insights-runtime-extractor-62sqq\" (UID: \"6e28953c-9a22-4850-941c-e9aabd07aef4\") " pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:01.903527 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:01.903468    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4z5\" (UniqueName: \"kubernetes.io/projected/6e28953c-9a22-4850-941c-e9aabd07aef4-kube-api-access-zw4z5\") pod \"insights-runtime-extractor-62sqq\" (UID: \"6e28953c-9a22-4850-941c-e9aabd07aef4\") " pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:01.903527 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:01.903489    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e28953c-9a22-4850-941c-e9aabd07aef4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-62sqq\" (UID: \"6e28953c-9a22-4850-941c-e9aabd07aef4\") " pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:02.004222 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:02.004135    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e28953c-9a22-4850-941c-e9aabd07aef4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-62sqq\" (UID: \"6e28953c-9a22-4850-941c-e9aabd07aef4\") " pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:02.004222 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:02.004179    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e28953c-9a22-4850-941c-e9aabd07aef4-crio-socket\") pod \"insights-runtime-extractor-62sqq\" (UID: \"6e28953c-9a22-4850-941c-e9aabd07aef4\") " pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:02.004428 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:02.004241    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e28953c-9a22-4850-941c-e9aabd07aef4-data-volume\") pod \"insights-runtime-extractor-62sqq\" (UID: \"6e28953c-9a22-4850-941c-e9aabd07aef4\") " pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:02.004428 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:02.004278    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4z5\" (UniqueName: \"kubernetes.io/projected/6e28953c-9a22-4850-941c-e9aabd07aef4-kube-api-access-zw4z5\") pod \"insights-runtime-extractor-62sqq\" (UID: \"6e28953c-9a22-4850-941c-e9aabd07aef4\") " pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:02.004428 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:02.004317    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e28953c-9a22-4850-941c-e9aabd07aef4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-62sqq\" (UID: \"6e28953c-9a22-4850-941c-e9aabd07aef4\") " pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:02.004428 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:02.004314    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e28953c-9a22-4850-941c-e9aabd07aef4-crio-socket\") pod \"insights-runtime-extractor-62sqq\" (UID: \"6e28953c-9a22-4850-941c-e9aabd07aef4\") " pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:02.004704 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:02.004678    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e28953c-9a22-4850-941c-e9aabd07aef4-data-volume\") pod \"insights-runtime-extractor-62sqq\" (UID: \"6e28953c-9a22-4850-941c-e9aabd07aef4\") " pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:02.004774 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:02.004756    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e28953c-9a22-4850-941c-e9aabd07aef4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-62sqq\" (UID: \"6e28953c-9a22-4850-941c-e9aabd07aef4\") " pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:02.006575 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:02.006557    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e28953c-9a22-4850-941c-e9aabd07aef4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-62sqq\" (UID: \"6e28953c-9a22-4850-941c-e9aabd07aef4\") " pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:02.019206 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:02.019183    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4z5\" (UniqueName: \"kubernetes.io/projected/6e28953c-9a22-4850-941c-e9aabd07aef4-kube-api-access-zw4z5\") pod \"insights-runtime-extractor-62sqq\" (UID: \"6e28953c-9a22-4850-941c-e9aabd07aef4\") " pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:02.064712 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:02.064676    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-62sqq"
May 28 00:14:02.186720 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:02.186697    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-62sqq"]
May 28 00:14:02.189349 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:14:02.189314    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e28953c_9a22_4850_941c_e9aabd07aef4.slice/crio-58a1ff571bdc8e14ca44f05a59e0fde8660798a33bda70fcdecdcba55a69880a WatchSource:0}: Error finding container 58a1ff571bdc8e14ca44f05a59e0fde8660798a33bda70fcdecdcba55a69880a: Status 404 returned error can't find the container with id 58a1ff571bdc8e14ca44f05a59e0fde8660798a33bda70fcdecdcba55a69880a
May 28 00:14:02.449632 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:02.449601    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-62sqq" event={"ID":"6e28953c-9a22-4850-941c-e9aabd07aef4","Type":"ContainerStarted","Data":"bb4d706ab550c7aa3f12622d2b194fdc5e7ff698b3299781f6394631b556bd14"}
May 28 00:14:02.449800 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:02.449640    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-62sqq" event={"ID":"6e28953c-9a22-4850-941c-e9aabd07aef4","Type":"ContainerStarted","Data":"58a1ff571bdc8e14ca44f05a59e0fde8660798a33bda70fcdecdcba55a69880a"}
May 28 00:14:03.211695 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.211655    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:14:03.214072 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.214043    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls\") pod \"image-registry-79b94db5c6-b4nm5\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") " pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:14:03.241168 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.241141    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-srcnj\""
May 28 00:14:03.249263 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.249244    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:14:03.312581 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.312555    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:14:03.312701 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.312596    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert\") pod \"ingress-canary-tgwt7\" (UID: \"b1715227-4b2d-44dd-a153-51973245938a\") " pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:14:03.314850 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.314827    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e2c7ab9-c05a-44d8-907a-377754c23b53-metrics-tls\") pod \"dns-default-4sq87\" (UID: \"2e2c7ab9-c05a-44d8-907a-377754c23b53\") " pod="openshift-dns/dns-default-4sq87"
May 28 00:14:03.314980 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.314961    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1715227-4b2d-44dd-a153-51973245938a-cert\") pod \"ingress-canary-tgwt7\" (UID: \"b1715227-4b2d-44dd-a153-51973245938a\") " pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:14:03.374393 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.374358    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79b94db5c6-b4nm5"]
May 28 00:14:03.377245 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:14:03.377220    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0045af_7f37_4ae7_89ec_7c66c799c21c.slice/crio-0e97ef78d3d31c9478a5663a80fca3f20794f29bea4573c0c97766e7bc5583c1 WatchSource:0}: Error finding container 0e97ef78d3d31c9478a5663a80fca3f20794f29bea4573c0c97766e7bc5583c1: Status 404 returned error can't find the container with id 0e97ef78d3d31c9478a5663a80fca3f20794f29bea4573c0c97766e7bc5583c1
May 28 00:14:03.456162 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.456132    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5" event={"ID":"0a0045af-7f37-4ae7-89ec-7c66c799c21c","Type":"ContainerStarted","Data":"43b2b6b8c1b8a579f5a40b561118f68e56fb6121d5271dac947ab9f7c880f36f"}
May 28 00:14:03.456304 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.456169    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5" event={"ID":"0a0045af-7f37-4ae7-89ec-7c66c799c21c","Type":"ContainerStarted","Data":"0e97ef78d3d31c9478a5663a80fca3f20794f29bea4573c0c97766e7bc5583c1"}
May 28 00:14:03.456304 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.456231    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:14:03.457690 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.457668    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-62sqq" event={"ID":"6e28953c-9a22-4850-941c-e9aabd07aef4","Type":"ContainerStarted","Data":"19e3a65255719ae66dde275c1eedb88cb828b34ee37ebd49d763d8a7c5da003f"}
May 28 00:14:03.477608 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.477567    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5" podStartSLOduration=162.477554761 podStartE2EDuration="2m42.477554761s" podCreationTimestamp="2026-05-28 00:11:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:14:03.476671916 +0000 UTC m=+163.097703928" watchObservedRunningTime="2026-05-28 00:14:03.477554761 +0000 UTC m=+163.098586805"
May 28 00:14:03.540605 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.540569    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nx64p\""
May 28 00:14:03.540767 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.540580    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzrg4\""
May 28 00:14:03.548718 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.548652    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tgwt7"
May 28 00:14:03.548818 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.548741    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4sq87"
May 28 00:14:03.704124 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.704085    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4sq87"]
May 28 00:14:03.705775 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:14:03.705747    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2c7ab9_c05a_44d8_907a_377754c23b53.slice/crio-63d8e71731d58d712e7727a1a29e1df16ab459fb3815c58e2a06628fa20b1f3c WatchSource:0}: Error finding container 63d8e71731d58d712e7727a1a29e1df16ab459fb3815c58e2a06628fa20b1f3c: Status 404 returned error can't find the container with id 63d8e71731d58d712e7727a1a29e1df16ab459fb3815c58e2a06628fa20b1f3c
May 28 00:14:03.717422 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:03.717394    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tgwt7"]
May 28 00:14:03.719938 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:14:03.719912    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1715227_4b2d_44dd_a153_51973245938a.slice/crio-7000b0050a16648e3c3311552afe6e613c738cb3573962dc024f0159f5d9198c WatchSource:0}: Error finding container 7000b0050a16648e3c3311552afe6e613c738cb3573962dc024f0159f5d9198c: Status 404 returned error can't find the container with id 7000b0050a16648e3c3311552afe6e613c738cb3573962dc024f0159f5d9198c
May 28 00:14:04.463010 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:04.462946    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4sq87" event={"ID":"2e2c7ab9-c05a-44d8-907a-377754c23b53","Type":"ContainerStarted","Data":"63d8e71731d58d712e7727a1a29e1df16ab459fb3815c58e2a06628fa20b1f3c"}
May 28 00:14:04.464539 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:04.464493    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tgwt7" event={"ID":"b1715227-4b2d-44dd-a153-51973245938a","Type":"ContainerStarted","Data":"7000b0050a16648e3c3311552afe6e613c738cb3573962dc024f0159f5d9198c"}
May 28 00:14:06.471446 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:06.471406    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4sq87" event={"ID":"2e2c7ab9-c05a-44d8-907a-377754c23b53","Type":"ContainerStarted","Data":"c1f6d03c9a6490713f5bc434a11a4d6fc767409fa95fb9941f9bcad318ca8e61"}
May 28 00:14:06.471446 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:06.471445    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4sq87" event={"ID":"2e2c7ab9-c05a-44d8-907a-377754c23b53","Type":"ContainerStarted","Data":"16a6cb07bb49d8abb045923eeb30cd790c200e54da136b175d5af91a1382937e"}
May 28 00:14:06.471997 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:06.471481    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4sq87"
May 28 00:14:06.473132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:06.473091    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-62sqq" event={"ID":"6e28953c-9a22-4850-941c-e9aabd07aef4","Type":"ContainerStarted","Data":"b6a0029f2983f2c2e0339407664e4a719866b2273b351358c4f14786a85530c3"}
May 28 00:14:06.474277 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:06.474258    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tgwt7" event={"ID":"b1715227-4b2d-44dd-a153-51973245938a","Type":"ContainerStarted","Data":"4509e2b10476af0dd910ea528ec4834a6e31ffa4a632b5bdfb0cfa73ef3894a5"}
May 28 00:14:06.489571 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:06.489528    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-4sq87" podStartSLOduration=129.419507473 podStartE2EDuration="2m11.489497626s" podCreationTimestamp="2026-05-28 00:11:55 +0000 UTC" firstStartedPulling="2026-05-28 00:14:03.707597604 +0000 UTC m=+163.328629595" lastFinishedPulling="2026-05-28 00:14:05.777587742 +0000 UTC m=+165.398619748" observedRunningTime="2026-05-28 00:14:06.489186776 +0000 UTC m=+166.110218789" watchObservedRunningTime="2026-05-28 00:14:06.489497626 +0000 UTC m=+166.110529660"
May 28 00:14:06.505530 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:06.505462    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tgwt7" podStartSLOduration=129.446249981 podStartE2EDuration="2m11.505443425s" podCreationTimestamp="2026-05-28 00:11:55 +0000 UTC" firstStartedPulling="2026-05-28 00:14:03.721831383 +0000 UTC m=+163.342863373" lastFinishedPulling="2026-05-28 00:14:05.781024818 +0000 UTC m=+165.402056817" observedRunningTime="2026-05-28 00:14:06.505057386 +0000 UTC m=+166.126089402" watchObservedRunningTime="2026-05-28 00:14:06.505443425 +0000 UTC m=+166.126475440"
May 28 00:14:06.522677 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:06.522623    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-62sqq" podStartSLOduration=1.99323621 podStartE2EDuration="5.522605463s" podCreationTimestamp="2026-05-28 00:14:01 +0000 UTC" firstStartedPulling="2026-05-28 00:14:02.247063284 +0000 UTC m=+161.868095275" lastFinishedPulling="2026-05-28 00:14:05.776432526 +0000 UTC m=+165.397464528" observedRunningTime="2026-05-28 00:14:06.521723303 +0000 UTC m=+166.142755341" watchObservedRunningTime="2026-05-28 00:14:06.522605463 +0000 UTC m=+166.143637478"
May 28 00:14:06.541226 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:06.541191    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bpdzw\" (UID: \"3c036ea7-0bb5-45d4-bacb-3b17d1156f0b\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"
May 28 00:14:06.543920 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:06.543886    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3c036ea7-0bb5-45d4-bacb-3b17d1156f0b-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bpdzw\" (UID: \"3c036ea7-0bb5-45d4-bacb-3b17d1156f0b\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"
May 28 00:14:06.802739 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:06.802701    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"
May 28 00:14:06.926805 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:06.926633    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw"]
May 28 00:14:06.929318 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:14:06.929286    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c036ea7_0bb5_45d4_bacb_3b17d1156f0b.slice/crio-d95543efbb40f766244f3a38b95f8fe793ad8d81f1a5846547a729aaf24ca0ec WatchSource:0}: Error finding container d95543efbb40f766244f3a38b95f8fe793ad8d81f1a5846547a729aaf24ca0ec: Status 404 returned error can't find the container with id d95543efbb40f766244f3a38b95f8fe793ad8d81f1a5846547a729aaf24ca0ec
May 28 00:14:07.424838 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.424801    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-cbd79fd76-ldsz7"]
May 28 00:14:07.429272 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.429249    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.433629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.433187    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7p86s\""
May 28 00:14:07.433629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.433487    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 28 00:14:07.433629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.433496    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 28 00:14:07.433832 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.433819    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 28 00:14:07.434093 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.434067    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 28 00:14:07.434417 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.434078    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 28 00:14:07.434709 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.434134    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 28 00:14:07.434969 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.434185    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 28 00:14:07.442089 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.442066    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbd79fd76-ldsz7"]
May 28 00:14:07.477586 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.477547    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw" event={"ID":"3c036ea7-0bb5-45d4-bacb-3b17d1156f0b","Type":"ContainerStarted","Data":"d95543efbb40f766244f3a38b95f8fe793ad8d81f1a5846547a729aaf24ca0ec"}
May 28 00:14:07.549647 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.549603    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-console-config\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.549819 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.549671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jqdl\" (UniqueName: \"kubernetes.io/projected/b6399827-e342-4eed-a849-f7e218bf6bcb-kube-api-access-4jqdl\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.549819 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.549723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-oauth-serving-cert\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.550083 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.550058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-service-ca\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.550182 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.550167    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6399827-e342-4eed-a849-f7e218bf6bcb-console-oauth-config\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.550290 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.550259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6399827-e342-4eed-a849-f7e218bf6bcb-console-serving-cert\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.650776 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.650735    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6399827-e342-4eed-a849-f7e218bf6bcb-console-oauth-config\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.650964 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.650793    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6399827-e342-4eed-a849-f7e218bf6bcb-console-serving-cert\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.650964 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.650828    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-console-config\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.650964 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.650857    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4jqdl\" (UniqueName: \"kubernetes.io/projected/b6399827-e342-4eed-a849-f7e218bf6bcb-kube-api-access-4jqdl\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.650964 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.650891    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-oauth-serving-cert\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.650964 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.650914    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-service-ca\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.651593 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.651569    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-console-config\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.651718 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.651665    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-service-ca\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.651766 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.651708    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-oauth-serving-cert\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.653242 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.653210    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6399827-e342-4eed-a849-f7e218bf6bcb-console-serving-cert\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.653333 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.653244    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6399827-e342-4eed-a849-f7e218bf6bcb-console-oauth-config\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.659801 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.659777    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jqdl\" (UniqueName: \"kubernetes.io/projected/b6399827-e342-4eed-a849-f7e218bf6bcb-kube-api-access-4jqdl\") pod \"console-cbd79fd76-ldsz7\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") " pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.742876 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.742788    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:07.882917 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:07.882891    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbd79fd76-ldsz7"]
May 28 00:14:07.885215 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:14:07.885186    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6399827_e342_4eed_a849_f7e218bf6bcb.slice/crio-230342f1065224f5c98e7120cdf23c31ce50f6eb9e3426cfb9932fbf899389dd WatchSource:0}: Error finding container 230342f1065224f5c98e7120cdf23c31ce50f6eb9e3426cfb9932fbf899389dd: Status 404 returned error can't find the container with id 230342f1065224f5c98e7120cdf23c31ce50f6eb9e3426cfb9932fbf899389dd
May 28 00:14:08.481561 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:08.481524    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbd79fd76-ldsz7" event={"ID":"b6399827-e342-4eed-a849-f7e218bf6bcb","Type":"ContainerStarted","Data":"230342f1065224f5c98e7120cdf23c31ce50f6eb9e3426cfb9932fbf899389dd"}
May 28 00:14:08.482761 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:08.482732    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw" event={"ID":"3c036ea7-0bb5-45d4-bacb-3b17d1156f0b","Type":"ContainerStarted","Data":"5cde035a809a3e8ae7a6f98032d6a36d8b63693ccc1db19b7df5a6c134e4bc77"}
May 28 00:14:08.499709 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:08.499660    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bpdzw" podStartSLOduration=33.06040689 podStartE2EDuration="34.49964558s" podCreationTimestamp="2026-05-28 00:13:34 +0000 UTC" firstStartedPulling="2026-05-28 00:14:06.931308384 +0000 UTC m=+166.552340388" lastFinishedPulling="2026-05-28 00:14:08.370547073 +0000 UTC m=+167.991579078" observedRunningTime="2026-05-28 00:14:08.499369374 +0000 UTC m=+168.120401408" watchObservedRunningTime="2026-05-28 00:14:08.49964558 +0000 UTC m=+168.120677642"
May 28 00:14:09.516586 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:09.516552    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-4d48s"]
May 28 00:14:09.521622 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:09.521599    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-4d48s"
May 28 00:14:09.525003 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:09.524740    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 28 00:14:09.526144 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:09.526119    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6mklb\""
May 28 00:14:09.526579 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:09.526555    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-4d48s"]
May 28 00:14:09.667092 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:09.667055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bd375ef7-7c1e-4074-8e93-6ac8f1d9920f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-4d48s\" (UID: \"bd375ef7-7c1e-4074-8e93-6ac8f1d9920f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-4d48s"
May 28 00:14:09.768060 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:09.767974    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bd375ef7-7c1e-4074-8e93-6ac8f1d9920f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-4d48s\" (UID: \"bd375ef7-7c1e-4074-8e93-6ac8f1d9920f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-4d48s"
May 28 00:14:09.770389 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:09.770358    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bd375ef7-7c1e-4074-8e93-6ac8f1d9920f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-4d48s\" (UID: \"bd375ef7-7c1e-4074-8e93-6ac8f1d9920f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-4d48s"
May 28 00:14:09.833582 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:09.833545    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-4d48s"
May 28 00:14:09.962088 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:09.962053    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-4d48s"]
May 28 00:14:09.965194 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:14:09.965154    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd375ef7_7c1e_4074_8e93_6ac8f1d9920f.slice/crio-b440f76b103518700164084102b31dabdc016c7ae46492bf8c6ff8c481ee2dba WatchSource:0}: Error finding container b440f76b103518700164084102b31dabdc016c7ae46492bf8c6ff8c481ee2dba: Status 404 returned error can't find the container with id b440f76b103518700164084102b31dabdc016c7ae46492bf8c6ff8c481ee2dba
May 28 00:14:10.491146 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:10.491073    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-4d48s" event={"ID":"bd375ef7-7c1e-4074-8e93-6ac8f1d9920f","Type":"ContainerStarted","Data":"b440f76b103518700164084102b31dabdc016c7ae46492bf8c6ff8c481ee2dba"}
May 28 00:14:12.011496 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:12.011454    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:14:13.500630 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:13.500593    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-4d48s" event={"ID":"bd375ef7-7c1e-4074-8e93-6ac8f1d9920f","Type":"ContainerStarted","Data":"9bae079105c2b2647d42903ea185a814450a9030f2ca95202590ea480be34a0b"}
May 28 00:14:13.501141 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:13.500847    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-4d48s"
May 28 00:14:13.502209 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:13.502174    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbd79fd76-ldsz7" event={"ID":"b6399827-e342-4eed-a849-f7e218bf6bcb","Type":"ContainerStarted","Data":"936aef59be9f45617cd6e5a58e6507c2f9dbb6aca400b309d575607e2aaf9a9c"}
May 28 00:14:13.506391 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:13.506371    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-4d48s"
May 28 00:14:13.518006 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:13.517966    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-4d48s" podStartSLOduration=1.7017254309999998 podStartE2EDuration="4.517954592s" podCreationTimestamp="2026-05-28 00:14:09 +0000 UTC" firstStartedPulling="2026-05-28 00:14:09.967085463 +0000 UTC m=+169.588117454" lastFinishedPulling="2026-05-28 00:14:12.783314612 +0000 UTC m=+172.404346615" observedRunningTime="2026-05-28 00:14:13.516144465 +0000 UTC m=+173.137176478" watchObservedRunningTime="2026-05-28 00:14:13.517954592 +0000 UTC m=+173.138986627"
May 28 00:14:13.532771 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:13.532724    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-cbd79fd76-ldsz7" podStartSLOduration=1.6302931379999999 podStartE2EDuration="6.532709371s" podCreationTimestamp="2026-05-28 00:14:07 +0000 UTC" firstStartedPulling="2026-05-28 00:14:07.887076081 +0000 UTC m=+167.508108072" lastFinishedPulling="2026-05-28 00:14:12.789492311 +0000 UTC m=+172.410524305" observedRunningTime="2026-05-28 00:14:13.532048204 +0000 UTC m=+173.153080229" watchObservedRunningTime="2026-05-28 00:14:13.532709371 +0000 UTC m=+173.153741384"
May 28 00:14:14.584443 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.584402    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"]
May 28 00:14:14.589274 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.589251    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"
May 28 00:14:14.593244 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.593199    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 28 00:14:14.593244 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.593221    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 28 00:14:14.593244 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.593230    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 28 00:14:14.593523 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.593261    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 28 00:14:14.593523 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.593199    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-g46n4\""
May 28 00:14:14.593901 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.593203    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 28 00:14:14.598081 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.598060    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"]
May 28 00:14:14.711346 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.711309    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dcnt\" (UniqueName: \"kubernetes.io/projected/03f56a8b-5149-4ba6-a9a5-eb14ce9edf74-kube-api-access-2dcnt\") pod \"prometheus-operator-7779bbdd78-sjmrd\" (UID: \"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"
May 28 00:14:14.711544 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.711387    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03f56a8b-5149-4ba6-a9a5-eb14ce9edf74-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-sjmrd\" (UID: \"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"
May 28 00:14:14.711544 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.711416    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03f56a8b-5149-4ba6-a9a5-eb14ce9edf74-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-sjmrd\" (UID: \"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"
May 28 00:14:14.711544 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.711450    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/03f56a8b-5149-4ba6-a9a5-eb14ce9edf74-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-sjmrd\" (UID: \"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"
May 28 00:14:14.812674 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.812635    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03f56a8b-5149-4ba6-a9a5-eb14ce9edf74-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-sjmrd\" (UID: \"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"
May 28 00:14:14.812841 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.812692    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/03f56a8b-5149-4ba6-a9a5-eb14ce9edf74-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-sjmrd\" (UID: \"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"
May 28 00:14:14.812841 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.812726    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2dcnt\" (UniqueName: \"kubernetes.io/projected/03f56a8b-5149-4ba6-a9a5-eb14ce9edf74-kube-api-access-2dcnt\") pod \"prometheus-operator-7779bbdd78-sjmrd\" (UID: \"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"
May 28 00:14:14.812841 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.812779    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03f56a8b-5149-4ba6-a9a5-eb14ce9edf74-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-sjmrd\" (UID: \"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"
May 28 00:14:14.812952 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:14:14.812871    2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
May 28 00:14:14.812952 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:14:14.812944    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f56a8b-5149-4ba6-a9a5-eb14ce9edf74-prometheus-operator-tls podName:03f56a8b-5149-4ba6-a9a5-eb14ce9edf74 nodeName:}" failed. No retries permitted until 2026-05-28 00:14:15.312924984 +0000 UTC m=+174.933956981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/03f56a8b-5149-4ba6-a9a5-eb14ce9edf74-prometheus-operator-tls") pod "prometheus-operator-7779bbdd78-sjmrd" (UID: "03f56a8b-5149-4ba6-a9a5-eb14ce9edf74") : secret "prometheus-operator-tls" not found
May 28 00:14:14.813464 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.813441    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03f56a8b-5149-4ba6-a9a5-eb14ce9edf74-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-sjmrd\" (UID: \"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"
May 28 00:14:14.815102 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.815079    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03f56a8b-5149-4ba6-a9a5-eb14ce9edf74-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-sjmrd\" (UID: \"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"
May 28 00:14:14.821699 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:14.821679    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dcnt\" (UniqueName: \"kubernetes.io/projected/03f56a8b-5149-4ba6-a9a5-eb14ce9edf74-kube-api-access-2dcnt\") pod \"prometheus-operator-7779bbdd78-sjmrd\" (UID: \"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"
May 28 00:14:15.317870 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:15.317820    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/03f56a8b-5149-4ba6-a9a5-eb14ce9edf74-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-sjmrd\" (UID: \"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"
May 28 00:14:15.320285 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:15.320263    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/03f56a8b-5149-4ba6-a9a5-eb14ce9edf74-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-sjmrd\" (UID: \"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"
May 28 00:14:15.499004 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:15.498965    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"
May 28 00:14:15.622431 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:15.622396    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd"]
May 28 00:14:15.625477 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:14:15.625447    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f56a8b_5149_4ba6_a9a5_eb14ce9edf74.slice/crio-071f3fd7ce3e933988dd66b2aa6949dca359298a39e8393932d54de0d5fd6701 WatchSource:0}: Error finding container 071f3fd7ce3e933988dd66b2aa6949dca359298a39e8393932d54de0d5fd6701: Status 404 returned error can't find the container with id 071f3fd7ce3e933988dd66b2aa6949dca359298a39e8393932d54de0d5fd6701
May 28 00:14:16.479550 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:16.479521    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4sq87"
May 28 00:14:16.512876 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:16.512847    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd" event={"ID":"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74","Type":"ContainerStarted","Data":"071f3fd7ce3e933988dd66b2aa6949dca359298a39e8393932d54de0d5fd6701"}
May 28 00:14:17.516970 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:17.516883    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd" event={"ID":"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74","Type":"ContainerStarted","Data":"df53ca52b04c033d647767f7ccfc51df618e194a7f9eb0a2d4f8e273cb994569"}
May 28 00:14:17.516970 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:17.516922    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd" event={"ID":"03f56a8b-5149-4ba6-a9a5-eb14ce9edf74","Type":"ContainerStarted","Data":"dfe70a684381185f6bb4c31d6c7c83e42c4a395f3dc496e525bcfdb16802503b"}
May 28 00:14:17.543157 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:17.543104    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-sjmrd" podStartSLOduration=1.991131843 podStartE2EDuration="3.543089468s" podCreationTimestamp="2026-05-28 00:14:14 +0000 UTC" firstStartedPulling="2026-05-28 00:14:15.627356046 +0000 UTC m=+175.248388037" lastFinishedPulling="2026-05-28 00:14:17.179313667 +0000 UTC m=+176.800345662" observedRunningTime="2026-05-28 00:14:17.543045406 +0000 UTC m=+177.164077420" watchObservedRunningTime="2026-05-28 00:14:17.543089468 +0000 UTC m=+177.164121480"
May 28 00:14:17.743288 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:17.743249    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:17.743448 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:17.743304    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:17.744482 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:17.744453    2576 patch_prober.go:28] interesting pod/console-cbd79fd76-ldsz7 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.18:8443/health\": dial tcp 10.132.0.18:8443: connect: connection refused" start-of-body=
May 28 00:14:17.744555 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:17.744524    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-cbd79fd76-ldsz7" podUID="b6399827-e342-4eed-a849-f7e218bf6bcb" containerName="console" probeResult="failure" output="Get \"https://10.132.0.18:8443/health\": dial tcp 10.132.0.18:8443: connect: connection refused"
May 28 00:14:19.984963 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:19.984923    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"]
May 28 00:14:19.990178 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:19.990154    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"
May 28 00:14:19.994082 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:19.994061    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 28 00:14:19.994403 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:19.994141    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 28 00:14:19.995998 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:19.995977    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-c2gd4\""
May 28 00:14:19.998053 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:19.998028    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"]
May 28 00:14:20.003876 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.003849    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"]
May 28 00:14:20.009038 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.009013    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.015290 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.015264    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-t5k6t\""
May 28 00:14:20.015598 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.015576    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 28 00:14:20.015738 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.015627    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 28 00:14:20.015799 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.015690    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 28 00:14:20.025714 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.025682    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"]
May 28 00:14:20.029320 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.029294    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9lwc6"]
May 28 00:14:20.034941 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.034921    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.038312 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.038293    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9rdvf\""
May 28 00:14:20.038561 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.038546    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 28 00:14:20.038697 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.038654    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 28 00:14:20.038780 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.038618    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 28 00:14:20.161100 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2e9c810f-5c20-42f3-93a9-583827884992-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.161284 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161112    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ef743b7-db78-4328-901c-1547852b6226-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.161284 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161192    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lfmh\" (UniqueName: \"kubernetes.io/projected/2e9c810f-5c20-42f3-93a9-583827884992-kube-api-access-7lfmh\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.161396 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161301    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e9c810f-5c20-42f3-93a9-583827884992-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.161396 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ef743b7-db78-4328-901c-1547852b6226-node-exporter-wtmp\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.161396 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vs2s\" (UniqueName: \"kubernetes.io/projected/2ef743b7-db78-4328-901c-1547852b6226-kube-api-access-4vs2s\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.161549 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161420    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ef743b7-db78-4328-901c-1547852b6226-root\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.161549 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161445    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ef743b7-db78-4328-901c-1547852b6226-node-exporter-textfile\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.161549 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9c810f-5c20-42f3-93a9-583827884992-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.161549 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpkgv\" (UniqueName: \"kubernetes.io/projected/233f08a0-994d-42e8-9777-d2f4f17ba908-kube-api-access-cpkgv\") pod \"openshift-state-metrics-9c959559f-b9z6f\" (UID: \"233f08a0-994d-42e8-9777-d2f4f17ba908\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"
May 28 00:14:20.161751 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161632    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ef743b7-db78-4328-901c-1547852b6226-metrics-client-ca\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.161751 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2e9c810f-5c20-42f3-93a9-583827884992-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.161751 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e9c810f-5c20-42f3-93a9-583827884992-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.161903 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161791    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/233f08a0-994d-42e8-9777-d2f4f17ba908-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-b9z6f\" (UID: \"233f08a0-994d-42e8-9777-d2f4f17ba908\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"
May 28 00:14:20.161903 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/233f08a0-994d-42e8-9777-d2f4f17ba908-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-b9z6f\" (UID: \"233f08a0-994d-42e8-9777-d2f4f17ba908\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"
May 28 00:14:20.161996 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ef743b7-db78-4328-901c-1547852b6226-sys\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.161996 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ef743b7-db78-4328-901c-1547852b6226-node-exporter-tls\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.161996 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161963    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ef743b7-db78-4328-901c-1547852b6226-node-exporter-accelerators-collector-config\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.161996 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.161992    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/233f08a0-994d-42e8-9777-d2f4f17ba908-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-b9z6f\" (UID: \"233f08a0-994d-42e8-9777-d2f4f17ba908\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"
May 28 00:14:20.263185 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263101    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/233f08a0-994d-42e8-9777-d2f4f17ba908-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-b9z6f\" (UID: \"233f08a0-994d-42e8-9777-d2f4f17ba908\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"
May 28 00:14:20.263185 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263147    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/233f08a0-994d-42e8-9777-d2f4f17ba908-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-b9z6f\" (UID: \"233f08a0-994d-42e8-9777-d2f4f17ba908\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"
May 28 00:14:20.263185 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263179    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ef743b7-db78-4328-901c-1547852b6226-sys\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.263453 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263202    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ef743b7-db78-4328-901c-1547852b6226-node-exporter-tls\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.263453 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263260    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ef743b7-db78-4328-901c-1547852b6226-node-exporter-accelerators-collector-config\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.263453 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263290    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/233f08a0-994d-42e8-9777-d2f4f17ba908-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-b9z6f\" (UID: \"233f08a0-994d-42e8-9777-d2f4f17ba908\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"
May 28 00:14:20.263453 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263313    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ef743b7-db78-4328-901c-1547852b6226-sys\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.263453 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263321    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2e9c810f-5c20-42f3-93a9-583827884992-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.263453 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263387    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ef743b7-db78-4328-901c-1547852b6226-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.263453 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263428    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7lfmh\" (UniqueName: \"kubernetes.io/projected/2e9c810f-5c20-42f3-93a9-583827884992-kube-api-access-7lfmh\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.263807 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263482    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e9c810f-5c20-42f3-93a9-583827884992-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.263807 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263544    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ef743b7-db78-4328-901c-1547852b6226-node-exporter-wtmp\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.263807 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263571    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4vs2s\" (UniqueName: \"kubernetes.io/projected/2ef743b7-db78-4328-901c-1547852b6226-kube-api-access-4vs2s\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.263807 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263602    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ef743b7-db78-4328-901c-1547852b6226-root\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.263807 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263626    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ef743b7-db78-4328-901c-1547852b6226-node-exporter-textfile\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.263807 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263670    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9c810f-5c20-42f3-93a9-583827884992-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.263807 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263698    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cpkgv\" (UniqueName: \"kubernetes.io/projected/233f08a0-994d-42e8-9777-d2f4f17ba908-kube-api-access-cpkgv\") pod \"openshift-state-metrics-9c959559f-b9z6f\" (UID: \"233f08a0-994d-42e8-9777-d2f4f17ba908\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"
May 28 00:14:20.263807 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263749    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ef743b7-db78-4328-901c-1547852b6226-metrics-client-ca\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.263807 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263806    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2e9c810f-5c20-42f3-93a9-583827884992-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.264238 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263836    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e9c810f-5c20-42f3-93a9-583827884992-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.264238 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.263996    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2e9c810f-5c20-42f3-93a9-583827884992-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.264343 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.264272    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e9c810f-5c20-42f3-93a9-583827884992-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.264479 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.264443    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ef743b7-db78-4328-901c-1547852b6226-node-exporter-wtmp\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.264562 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.264536    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ef743b7-db78-4328-901c-1547852b6226-root\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.264618 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.264574    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ef743b7-db78-4328-901c-1547852b6226-node-exporter-textfile\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.264797 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.264772    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ef743b7-db78-4328-901c-1547852b6226-node-exporter-accelerators-collector-config\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.265297 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.265251    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2e9c810f-5c20-42f3-93a9-583827884992-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.265659 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.265633    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/233f08a0-994d-42e8-9777-d2f4f17ba908-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-b9z6f\" (UID: \"233f08a0-994d-42e8-9777-d2f4f17ba908\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"
May 28 00:14:20.265966 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.265943    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ef743b7-db78-4328-901c-1547852b6226-metrics-client-ca\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.267392 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.267325    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e9c810f-5c20-42f3-93a9-583827884992-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.267751 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.267732    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ef743b7-db78-4328-901c-1547852b6226-node-exporter-tls\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.268432 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.268411    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/233f08a0-994d-42e8-9777-d2f4f17ba908-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-b9z6f\" (UID: \"233f08a0-994d-42e8-9777-d2f4f17ba908\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"
May 28 00:14:20.269567 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.269520    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9c810f-5c20-42f3-93a9-583827884992-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.269915 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.269889    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/233f08a0-994d-42e8-9777-d2f4f17ba908-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-b9z6f\" (UID: \"233f08a0-994d-42e8-9777-d2f4f17ba908\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"
May 28 00:14:20.270562 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.270543    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ef743b7-db78-4328-901c-1547852b6226-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.272574 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.272550    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lfmh\" (UniqueName: \"kubernetes.io/projected/2e9c810f-5c20-42f3-93a9-583827884992-kube-api-access-7lfmh\") pod \"kube-state-metrics-54fd6bc5c4-mj29t\" (UID: \"2e9c810f-5c20-42f3-93a9-583827884992\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.273410 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.273379    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vs2s\" (UniqueName: \"kubernetes.io/projected/2ef743b7-db78-4328-901c-1547852b6226-kube-api-access-4vs2s\") pod \"node-exporter-9lwc6\" (UID: \"2ef743b7-db78-4328-901c-1547852b6226\") " pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.274048 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.274028    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpkgv\" (UniqueName: \"kubernetes.io/projected/233f08a0-994d-42e8-9777-d2f4f17ba908-kube-api-access-cpkgv\") pod \"openshift-state-metrics-9c959559f-b9z6f\" (UID: \"233f08a0-994d-42e8-9777-d2f4f17ba908\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"
May 28 00:14:20.301893 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.301861    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"
May 28 00:14:20.319849 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.319819    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"
May 28 00:14:20.361716 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.361683    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9lwc6"
May 28 00:14:20.453242 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.453211    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f"]
May 28 00:14:20.455786 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:14:20.455758    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233f08a0_994d_42e8_9777_d2f4f17ba908.slice/crio-7954ffc5c23b3779598aace837546248678e4b3364c8203dc111c257b3e16493 WatchSource:0}: Error finding container 7954ffc5c23b3779598aace837546248678e4b3364c8203dc111c257b3e16493: Status 404 returned error can't find the container with id 7954ffc5c23b3779598aace837546248678e4b3364c8203dc111c257b3e16493
May 28 00:14:20.473024 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.473001    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t"]
May 28 00:14:20.475437 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:14:20.475410    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9c810f_5c20_42f3_93a9_583827884992.slice/crio-9d1f54c319c6579b239aca81606145aad5649b12ccc27647b75c8c1072c68190 WatchSource:0}: Error finding container 9d1f54c319c6579b239aca81606145aad5649b12ccc27647b75c8c1072c68190: Status 404 returned error can't find the container with id 9d1f54c319c6579b239aca81606145aad5649b12ccc27647b75c8c1072c68190
May 28 00:14:20.519663 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.519636    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cbd79fd76-ldsz7"]
May 28 00:14:20.525950 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.525920    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t" event={"ID":"2e9c810f-5c20-42f3-93a9-583827884992","Type":"ContainerStarted","Data":"9d1f54c319c6579b239aca81606145aad5649b12ccc27647b75c8c1072c68190"}
May 28 00:14:20.527806 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.527755    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lwc6" event={"ID":"2ef743b7-db78-4328-901c-1547852b6226","Type":"ContainerStarted","Data":"968701bcd2a5e9e48278756e633964dac4b6618227d15e8e19c6a9f253d0471b"}
May 28 00:14:20.529360 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.529331    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f" event={"ID":"233f08a0-994d-42e8-9777-d2f4f17ba908","Type":"ContainerStarted","Data":"02b1c54539d0faa64a165a7051e164f64c7b33f106536c09934e510e45872f59"}
May 28 00:14:20.529360 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:20.529363    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f" event={"ID":"233f08a0-994d-42e8-9777-d2f4f17ba908","Type":"ContainerStarted","Data":"7954ffc5c23b3779598aace837546248678e4b3364c8203dc111c257b3e16493"}
May 28 00:14:21.017543 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.017500    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 00:14:21.023211 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.023183    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.025825 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.025725    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 28 00:14:21.025825 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.025725    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rnmlr\""
May 28 00:14:21.025825 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.025799    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 28 00:14:21.025825 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.025748    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 28 00:14:21.026101 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.025998    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 28 00:14:21.026101 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.026083    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 28 00:14:21.026101 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.026089    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 28 00:14:21.026236 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.026187    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 28 00:14:21.026327 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.026311    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 28 00:14:21.026374 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.026366    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 28 00:14:21.034631 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.034604    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 00:14:21.172606 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.172568    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817606af-c726-4199-9fbe-6ce1ff5c6776-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.172790 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.172617    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-web-config\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.172790 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.172639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.172790 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.172740    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/817606af-c726-4199-9fbe-6ce1ff5c6776-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.173022 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.172789    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.173022 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.172881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-config-volume\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.173022 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.172926    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/817606af-c726-4199-9fbe-6ce1ff5c6776-tls-assets\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.173022 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.172951    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/817606af-c726-4199-9fbe-6ce1ff5c6776-config-out\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.173022 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.172977    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.173252 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.173031    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/817606af-c726-4199-9fbe-6ce1ff5c6776-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.173252 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.173070    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bzrp\" (UniqueName: \"kubernetes.io/projected/817606af-c726-4199-9fbe-6ce1ff5c6776-kube-api-access-8bzrp\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.173252 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.173126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.173406 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.173375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.273987 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.273901    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817606af-c726-4199-9fbe-6ce1ff5c6776-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.273987 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.273943    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-web-config\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.273987 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.273965    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.274316 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.273999    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/817606af-c726-4199-9fbe-6ce1ff5c6776-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.274316 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.274036    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.281654 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.275319    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817606af-c726-4199-9fbe-6ce1ff5c6776-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.281654 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.275402    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-config-volume\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.281654 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.275450    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/817606af-c726-4199-9fbe-6ce1ff5c6776-tls-assets\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.281654 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.275481    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/817606af-c726-4199-9fbe-6ce1ff5c6776-config-out\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.281654 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.275533    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.281654 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.275660    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/817606af-c726-4199-9fbe-6ce1ff5c6776-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.281654 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.275918    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/817606af-c726-4199-9fbe-6ce1ff5c6776-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.281654 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.275972    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8bzrp\" (UniqueName: \"kubernetes.io/projected/817606af-c726-4199-9fbe-6ce1ff5c6776-kube-api-access-8bzrp\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.281654 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.276019    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.281654 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.276071    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.281654 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.276855    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/817606af-c726-4199-9fbe-6ce1ff5c6776-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.281654 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.278549    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.288448 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.287635    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.288448 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.288006    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/817606af-c726-4199-9fbe-6ce1ff5c6776-tls-assets\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.288663 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.288461    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.288663 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.288560    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.289264 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.288954    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-web-config\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.289708 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.289652    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/817606af-c726-4199-9fbe-6ce1ff5c6776-config-out\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.290970 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.290689    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-config-volume\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.291073 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.291000    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bzrp\" (UniqueName: \"kubernetes.io/projected/817606af-c726-4199-9fbe-6ce1ff5c6776-kube-api-access-8bzrp\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.291356 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.291332    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/817606af-c726-4199-9fbe-6ce1ff5c6776-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"817606af-c726-4199-9fbe-6ce1ff5c6776\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.335154 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.334702    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 00:14:21.536751 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.536681    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f" event={"ID":"233f08a0-994d-42e8-9777-d2f4f17ba908","Type":"ContainerStarted","Data":"92744f0b9c7659925d2b659e0b500bf10dfaed8d493fdc99a36dccc75f2dc856"}
May 28 00:14:21.707468 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:21.707429    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 00:14:22.009312 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.009280    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-64f86464d4-qvwrw"]
May 28 00:14:22.013785 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.013763    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.016738 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.016713    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 28 00:14:22.016896 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.016868    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 28 00:14:22.017007 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.016941    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-479st\""
May 28 00:14:22.017007 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.016979    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 28 00:14:22.017007 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.016989    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 28 00:14:22.017209 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.017012    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 28 00:14:22.017209 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.017093    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-23faf2svfvcb5\""
May 28 00:14:22.022887 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.022863    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-64f86464d4-qvwrw"]
May 28 00:14:22.154152 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:14:22.154086    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817606af_c726_4199_9fbe_6ce1ff5c6776.slice/crio-d4d6fe5eb640dac0b5b320be0a8470cfbea57bdc7850c5706bb41ed11a3341b8 WatchSource:0}: Error finding container d4d6fe5eb640dac0b5b320be0a8470cfbea57bdc7850c5706bb41ed11a3341b8: Status 404 returned error can't find the container with id d4d6fe5eb640dac0b5b320be0a8470cfbea57bdc7850c5706bb41ed11a3341b8
May 28 00:14:22.184952 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.184916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.184952 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.184952    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz8r5\" (UniqueName: \"kubernetes.io/projected/fc9cc118-8d55-4701-b696-66e6f1816343-kube-api-access-lz8r5\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.185176 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.184990    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.185176 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.185061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc9cc118-8d55-4701-b696-66e6f1816343-metrics-client-ca\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.185176 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.185115    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.185176 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.185149    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-thanos-querier-tls\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.185313 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.185229    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.185313 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.185255    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-grpc-tls\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.286659 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.286622    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc9cc118-8d55-4701-b696-66e6f1816343-metrics-client-ca\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.286774 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.286678    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.286876 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.286856    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-thanos-querier-tls\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.286983 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.286953    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.287047 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.286985    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-grpc-tls\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.287103 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.287051    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.287103 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.287080    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lz8r5\" (UniqueName: \"kubernetes.io/projected/fc9cc118-8d55-4701-b696-66e6f1816343-kube-api-access-lz8r5\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.287197 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.287150    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.287589 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.287389    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc9cc118-8d55-4701-b696-66e6f1816343-metrics-client-ca\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.290631 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.290490    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.290631 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.290524    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.290796 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.290667    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-thanos-querier-tls\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.290796 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.290687    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.291871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.291821    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.291967 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.291894    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc9cc118-8d55-4701-b696-66e6f1816343-secret-grpc-tls\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.298230 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.298174    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz8r5\" (UniqueName: \"kubernetes.io/projected/fc9cc118-8d55-4701-b696-66e6f1816343-kube-api-access-lz8r5\") pod \"thanos-querier-64f86464d4-qvwrw\" (UID: \"fc9cc118-8d55-4701-b696-66e6f1816343\") " pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.326437 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.326350    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:22.493401 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.491574    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-64f86464d4-qvwrw"]
May 28 00:14:22.495782 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:14:22.495749    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9cc118_8d55_4701_b696_66e6f1816343.slice/crio-5eb5a0ae2b6a0d8154ee3e1ad5604f56620a4109954bf7b6c1c335c0d24b6df2 WatchSource:0}: Error finding container 5eb5a0ae2b6a0d8154ee3e1ad5604f56620a4109954bf7b6c1c335c0d24b6df2: Status 404 returned error can't find the container with id 5eb5a0ae2b6a0d8154ee3e1ad5604f56620a4109954bf7b6c1c335c0d24b6df2
May 28 00:14:22.540678 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.540642    2576 generic.go:358] "Generic (PLEG): container finished" podID="2ef743b7-db78-4328-901c-1547852b6226" containerID="bec2b1e79e98e62b6f31d86a76ada8d4e24759887c4afe503bbda8b8a65390b1" exitCode=0
May 28 00:14:22.540834 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.540725    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lwc6" event={"ID":"2ef743b7-db78-4328-901c-1547852b6226","Type":"ContainerDied","Data":"bec2b1e79e98e62b6f31d86a76ada8d4e24759887c4afe503bbda8b8a65390b1"}
May 28 00:14:22.541902 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.541875    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"817606af-c726-4199-9fbe-6ce1ff5c6776","Type":"ContainerStarted","Data":"d4d6fe5eb640dac0b5b320be0a8470cfbea57bdc7850c5706bb41ed11a3341b8"}
May 28 00:14:22.543886 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.543858    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f" event={"ID":"233f08a0-994d-42e8-9777-d2f4f17ba908","Type":"ContainerStarted","Data":"105015e9ff29a1734ac3d48d918d177941e5081637d79164b5518fab929471bd"}
May 28 00:14:22.545040 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.545020    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw" event={"ID":"fc9cc118-8d55-4701-b696-66e6f1816343","Type":"ContainerStarted","Data":"5eb5a0ae2b6a0d8154ee3e1ad5604f56620a4109954bf7b6c1c335c0d24b6df2"}
May 28 00:14:22.546922 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.546898    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t" event={"ID":"2e9c810f-5c20-42f3-93a9-583827884992","Type":"ContainerStarted","Data":"a946740722206b92115fd2ff43d38ce8b2a6ae10d3f9758b34cf0e40a3f08d1a"}
May 28 00:14:22.547011 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.546928    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t" event={"ID":"2e9c810f-5c20-42f3-93a9-583827884992","Type":"ContainerStarted","Data":"4ac49dc3fa62a603c403ef1fbfa7bb953692f6c50a03ae1c85fa01b6e2c036d5"}
May 28 00:14:22.547011 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.546943    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t" event={"ID":"2e9c810f-5c20-42f3-93a9-583827884992","Type":"ContainerStarted","Data":"5a52a770f9bb239169648d2c9d688a6d263abb0b6cde84ae94bcf66178791328"}
May 28 00:14:22.583794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.583735    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-mj29t" podStartSLOduration=1.851100582 podStartE2EDuration="3.583715854s" podCreationTimestamp="2026-05-28 00:14:19 +0000 UTC" firstStartedPulling="2026-05-28 00:14:20.477164738 +0000 UTC m=+180.098196730" lastFinishedPulling="2026-05-28 00:14:22.209780012 +0000 UTC m=+181.830812002" observedRunningTime="2026-05-28 00:14:22.581943742 +0000 UTC m=+182.202975779" watchObservedRunningTime="2026-05-28 00:14:22.583715854 +0000 UTC m=+182.204747865"
May 28 00:14:22.606366 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:22.606310    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-b9z6f" podStartSLOduration=1.984260512 podStartE2EDuration="3.606291582s" podCreationTimestamp="2026-05-28 00:14:19 +0000 UTC" firstStartedPulling="2026-05-28 00:14:20.584854659 +0000 UTC m=+180.205886655" lastFinishedPulling="2026-05-28 00:14:22.206885729 +0000 UTC m=+181.827917725" observedRunningTime="2026-05-28 00:14:22.605091829 +0000 UTC m=+182.226123843" watchObservedRunningTime="2026-05-28 00:14:22.606291582 +0000 UTC m=+182.227323598"
May 28 00:14:23.254489 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:23.254447    2576 patch_prober.go:28] interesting pod/image-registry-79b94db5c6-b4nm5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 28 00:14:23.254957 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:23.254545    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5" podUID="0a0045af-7f37-4ae7-89ec-7c66c799c21c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 28 00:14:23.552660 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:23.552620    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lwc6" event={"ID":"2ef743b7-db78-4328-901c-1547852b6226","Type":"ContainerStarted","Data":"48ce3eaadf861c9f8ebf140c46154649f601f46298640c220d475aca5e016b00"}
May 28 00:14:23.552835 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:23.552668    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lwc6" event={"ID":"2ef743b7-db78-4328-901c-1547852b6226","Type":"ContainerStarted","Data":"bcad46691a2ba51c8846b0ea253a1a6cff86ed8f49688c89777d48850a452965"}
May 28 00:14:23.576835 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:23.576781    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9lwc6" podStartSLOduration=1.753482708 podStartE2EDuration="3.576764653s" podCreationTimestamp="2026-05-28 00:14:20 +0000 UTC" firstStartedPulling="2026-05-28 00:14:20.379044927 +0000 UTC m=+180.000076921" lastFinishedPulling="2026-05-28 00:14:22.202326872 +0000 UTC m=+181.823358866" observedRunningTime="2026-05-28 00:14:23.574249213 +0000 UTC m=+183.195281239" watchObservedRunningTime="2026-05-28 00:14:23.576764653 +0000 UTC m=+183.197796665"
May 28 00:14:23.850075 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:23.849994    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79b94db5c6-b4nm5"]
May 28 00:14:23.855622 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:23.855540    2576 patch_prober.go:28] interesting pod/image-registry-79b94db5c6-b4nm5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 28 00:14:23.855622 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:23.855599    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5" podUID="0a0045af-7f37-4ae7-89ec-7c66c799c21c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 28 00:14:24.401829 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.401790    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-784984c5dc-64l2b"]
May 28 00:14:24.408125 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.408092    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.411448 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.411417    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 28 00:14:24.411767 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.411646    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 28 00:14:24.412662 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.412638    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 28 00:14:24.412776 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.412703    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lrwbb\""
May 28 00:14:24.412839 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.412798    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bu5tobjd4s85q\""
May 28 00:14:24.412897 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.412882    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 28 00:14:24.424528 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.424467    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-784984c5dc-64l2b"]
May 28 00:14:24.508525 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.508459    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-metrics-server-audit-profiles\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.508696 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.508532    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.508696 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.508643    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-client-ca-bundle\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.508804 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.508704    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-secret-metrics-server-tls\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.508804 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.508726    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-audit-log\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.508804 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.508756    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfz22\" (UniqueName: \"kubernetes.io/projected/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-kube-api-access-gfz22\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.508804 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.508796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-secret-metrics-server-client-certs\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.557948 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.557899    2576 generic.go:358] "Generic (PLEG): container finished" podID="817606af-c726-4199-9fbe-6ce1ff5c6776" containerID="826299e6868a6210cc6602649133bbb7e15c5ecc27c9e1541b5b804b436bd2aa" exitCode=0
May 28 00:14:24.558134 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.557985    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"817606af-c726-4199-9fbe-6ce1ff5c6776","Type":"ContainerDied","Data":"826299e6868a6210cc6602649133bbb7e15c5ecc27c9e1541b5b804b436bd2aa"}
May 28 00:14:24.609326 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.609286    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.609527 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.609355    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-client-ca-bundle\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.609527 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.609399    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-secret-metrics-server-tls\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.609527 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.609421    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-audit-log\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.609703 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.609562    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gfz22\" (UniqueName: \"kubernetes.io/projected/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-kube-api-access-gfz22\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.609703 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.609631    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-secret-metrics-server-client-certs\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.609795 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.609701    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-metrics-server-audit-profiles\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.610236 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.610201    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.610236 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.610208    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-audit-log\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.610740 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.610713    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-metrics-server-audit-profiles\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.612466 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.612418    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-client-ca-bundle\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.612604 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.612564    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-secret-metrics-server-tls\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.612693 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.612666    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-secret-metrics-server-client-certs\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.618993 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.618971    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfz22\" (UniqueName: \"kubernetes.io/projected/a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e-kube-api-access-gfz22\") pod \"metrics-server-784984c5dc-64l2b\" (UID: \"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e\") " pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:24.718964 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:24.718872    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:25.007887 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.007860    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-784984c5dc-64l2b"]
May 28 00:14:25.012035 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:14:25.012005    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b3729d_0bbc_42dc_a4b3_e0d6dbe2292e.slice/crio-23156ea35b2a8b676dbd3e847a76f781109f20594ee75053f587535feea10914 WatchSource:0}: Error finding container 23156ea35b2a8b676dbd3e847a76f781109f20594ee75053f587535feea10914: Status 404 returned error can't find the container with id 23156ea35b2a8b676dbd3e847a76f781109f20594ee75053f587535feea10914
May 28 00:14:25.174311 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.174249    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"]
May 28 00:14:25.178920 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.178895    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.182382 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.181759    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 28 00:14:25.182382 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.181776    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 28 00:14:25.182382 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.182201    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 28 00:14:25.182382 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.182379    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 28 00:14:25.182664 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.182470    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jhh4x\""
May 28 00:14:25.182664 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.182633    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 28 00:14:25.190209 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.190078    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 28 00:14:25.192649 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.192627    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"]
May 28 00:14:25.317953 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.317911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c46be0ad-baf5-4342-ad5d-407db2a83e0c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.318141 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.317965    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c46be0ad-baf5-4342-ad5d-407db2a83e0c-serving-certs-ca-bundle\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.318141 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.318025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c46be0ad-baf5-4342-ad5d-407db2a83e0c-telemeter-client-tls\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.318141 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.318073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c46be0ad-baf5-4342-ad5d-407db2a83e0c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.318141 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.318091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g9s7\" (UniqueName: \"kubernetes.io/projected/c46be0ad-baf5-4342-ad5d-407db2a83e0c-kube-api-access-9g9s7\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.318141 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.318109    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c46be0ad-baf5-4342-ad5d-407db2a83e0c-federate-client-tls\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.318141 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.318138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c46be0ad-baf5-4342-ad5d-407db2a83e0c-metrics-client-ca\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.318357 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.318238    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c46be0ad-baf5-4342-ad5d-407db2a83e0c-secret-telemeter-client\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.418966 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.418927    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c46be0ad-baf5-4342-ad5d-407db2a83e0c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.419537 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.418975    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9g9s7\" (UniqueName: \"kubernetes.io/projected/c46be0ad-baf5-4342-ad5d-407db2a83e0c-kube-api-access-9g9s7\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.419537 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.419003    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c46be0ad-baf5-4342-ad5d-407db2a83e0c-federate-client-tls\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.419537 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.419030    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c46be0ad-baf5-4342-ad5d-407db2a83e0c-metrics-client-ca\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.419537 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.419071    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c46be0ad-baf5-4342-ad5d-407db2a83e0c-secret-telemeter-client\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.419537 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.419106    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c46be0ad-baf5-4342-ad5d-407db2a83e0c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.419537 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.419144    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c46be0ad-baf5-4342-ad5d-407db2a83e0c-serving-certs-ca-bundle\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.419537 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.419187    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c46be0ad-baf5-4342-ad5d-407db2a83e0c-telemeter-client-tls\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.420028 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.419990    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c46be0ad-baf5-4342-ad5d-407db2a83e0c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.420088 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.420062    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c46be0ad-baf5-4342-ad5d-407db2a83e0c-metrics-client-ca\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.420659 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.420625    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c46be0ad-baf5-4342-ad5d-407db2a83e0c-serving-certs-ca-bundle\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.422287 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.422267    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c46be0ad-baf5-4342-ad5d-407db2a83e0c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.422287 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.422277    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c46be0ad-baf5-4342-ad5d-407db2a83e0c-secret-telemeter-client\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.422437 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.422372    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c46be0ad-baf5-4342-ad5d-407db2a83e0c-telemeter-client-tls\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.422550 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.422530    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c46be0ad-baf5-4342-ad5d-407db2a83e0c-federate-client-tls\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.427600 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.427577    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g9s7\" (UniqueName: \"kubernetes.io/projected/c46be0ad-baf5-4342-ad5d-407db2a83e0c-kube-api-access-9g9s7\") pod \"telemeter-client-7b5c989bd6-wz9zb\" (UID: \"c46be0ad-baf5-4342-ad5d-407db2a83e0c\") " pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.493229 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.493182    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"
May 28 00:14:25.563951 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.563917    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw" event={"ID":"fc9cc118-8d55-4701-b696-66e6f1816343","Type":"ContainerStarted","Data":"3950c9238407589895263bd75e9cfac3e45578c28479af1352bed54f751106c4"}
May 28 00:14:25.563951 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.563958    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw" event={"ID":"fc9cc118-8d55-4701-b696-66e6f1816343","Type":"ContainerStarted","Data":"782bf43dec6ee5cd5607acb7b4c56f2a2b3cec63c56907970eb270206b68b73f"}
May 28 00:14:25.564158 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.563971    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw" event={"ID":"fc9cc118-8d55-4701-b696-66e6f1816343","Type":"ContainerStarted","Data":"f2ababce00d2b8620266454d2e220b13ccb795328ef24c9b323ab198df854f41"}
May 28 00:14:25.565001 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.564967    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-784984c5dc-64l2b" event={"ID":"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e","Type":"ContainerStarted","Data":"23156ea35b2a8b676dbd3e847a76f781109f20594ee75053f587535feea10914"}
May 28 00:14:25.632482 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:25.632456    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb"]
May 28 00:14:25.634421 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:14:25.634390    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46be0ad_baf5_4342_ad5d_407db2a83e0c.slice/crio-5066c045937822bb0d8111b494c063ab7f3804a3a89ed68b9f771e51eae04283 WatchSource:0}: Error finding container 5066c045937822bb0d8111b494c063ab7f3804a3a89ed68b9f771e51eae04283: Status 404 returned error can't find the container with id 5066c045937822bb0d8111b494c063ab7f3804a3a89ed68b9f771e51eae04283
May 28 00:14:26.574831 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:26.574789    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw" event={"ID":"fc9cc118-8d55-4701-b696-66e6f1816343","Type":"ContainerStarted","Data":"8796b8fb635624c1195cb5105a6012e484ad58c5e4ea47fb67c80c3159b5b601"}
May 28 00:14:26.574831 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:26.574836    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw" event={"ID":"fc9cc118-8d55-4701-b696-66e6f1816343","Type":"ContainerStarted","Data":"7027895e6afab71a1a4c261cc2fd81d069fce6b3568096e81146dbfca47e3f2d"}
May 28 00:14:26.575254 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:26.574851    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw" event={"ID":"fc9cc118-8d55-4701-b696-66e6f1816343","Type":"ContainerStarted","Data":"0081f60afc534f3add910a61834a7402902e95e53e092e23f1e8c2179e40965f"}
May 28 00:14:26.575254 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:26.574972    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:26.576109 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:26.576086    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb" event={"ID":"c46be0ad-baf5-4342-ad5d-407db2a83e0c","Type":"ContainerStarted","Data":"5066c045937822bb0d8111b494c063ab7f3804a3a89ed68b9f771e51eae04283"}
May 28 00:14:26.602537 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:26.600000    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw" podStartSLOduration=1.868916162 podStartE2EDuration="5.599978841s" podCreationTimestamp="2026-05-28 00:14:21 +0000 UTC" firstStartedPulling="2026-05-28 00:14:22.498402531 +0000 UTC m=+182.119434522" lastFinishedPulling="2026-05-28 00:14:26.229465203 +0000 UTC m=+185.850497201" observedRunningTime="2026-05-28 00:14:26.598409403 +0000 UTC m=+186.219441419" watchObservedRunningTime="2026-05-28 00:14:26.599978841 +0000 UTC m=+186.221010855"
May 28 00:14:27.581481 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:27.581432    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-784984c5dc-64l2b" event={"ID":"a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e","Type":"ContainerStarted","Data":"b6ce8cc6015ba9c5a53bc3544f830f481254db0f969f80135a6b17922d70777a"}
May 28 00:14:27.583666 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:27.583628    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"817606af-c726-4199-9fbe-6ce1ff5c6776","Type":"ContainerStarted","Data":"30aafbeffda9ca9271baa96e679439bf87ed0b31e2b39f92e3c2957500af23a1"}
May 28 00:14:27.600659 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:27.600601    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-784984c5dc-64l2b" podStartSLOduration=1.252636953 podStartE2EDuration="3.600587253s" podCreationTimestamp="2026-05-28 00:14:24 +0000 UTC" firstStartedPulling="2026-05-28 00:14:25.020022554 +0000 UTC m=+184.641054558" lastFinishedPulling="2026-05-28 00:14:27.367972865 +0000 UTC m=+186.989004858" observedRunningTime="2026-05-28 00:14:27.598962311 +0000 UTC m=+187.219994324" watchObservedRunningTime="2026-05-28 00:14:27.600587253 +0000 UTC m=+187.221619266"
May 28 00:14:28.589133 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:28.589093    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb" event={"ID":"c46be0ad-baf5-4342-ad5d-407db2a83e0c","Type":"ContainerStarted","Data":"cb191875633b7f1e78ae66f4943e3e579efb587ab09348e9287629aaa457c5f3"}
May 28 00:14:28.589627 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:28.589140    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb" event={"ID":"c46be0ad-baf5-4342-ad5d-407db2a83e0c","Type":"ContainerStarted","Data":"bf44b261a1991f35e6fd59a434d67ebe666971b228532b538530e115d3b6b9fb"}
May 28 00:14:28.589627 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:28.589156    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb" event={"ID":"c46be0ad-baf5-4342-ad5d-407db2a83e0c","Type":"ContainerStarted","Data":"1e42abc5cfadc773eedfde25b81bc6114dc4fd39a11c823ecdc5479f90d4a41a"}
May 28 00:14:28.592176 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:28.592147    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"817606af-c726-4199-9fbe-6ce1ff5c6776","Type":"ContainerStarted","Data":"a7281822adfb69a78bfcf2c881086e7a25b1d6f4bacb155da161073a19c82736"}
May 28 00:14:28.592277 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:28.592184    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"817606af-c726-4199-9fbe-6ce1ff5c6776","Type":"ContainerStarted","Data":"339b5b7ff18686e0ce9e108abdabd9d105cd19a28880f68c1e87e9a2ccde879d"}
May 28 00:14:28.592277 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:28.592200    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"817606af-c726-4199-9fbe-6ce1ff5c6776","Type":"ContainerStarted","Data":"47229eb92ae04b3db145b4e5e89d9ffcbf9f225f0579c2c5beb0db31f70e7a24"}
May 28 00:14:28.592277 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:28.592212    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"817606af-c726-4199-9fbe-6ce1ff5c6776","Type":"ContainerStarted","Data":"b1624f51e6aa15a57917f21be9dca4182caeb4c3cdc351b041f4b0731f3b881c"}
May 28 00:14:28.592277 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:28.592225    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"817606af-c726-4199-9fbe-6ce1ff5c6776","Type":"ContainerStarted","Data":"8387d37870daa4ad1a78e3f7740b37a5d00209eee13ad07ab6b027db22bf7d14"}
May 28 00:14:28.612999 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:28.612957    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7b5c989bd6-wz9zb" podStartSLOduration=1.2981016699999999 podStartE2EDuration="3.612943072s" podCreationTimestamp="2026-05-28 00:14:25 +0000 UTC" firstStartedPulling="2026-05-28 00:14:25.636808891 +0000 UTC m=+185.257840884" lastFinishedPulling="2026-05-28 00:14:27.951650291 +0000 UTC m=+187.572682286" observedRunningTime="2026-05-28 00:14:28.612719613 +0000 UTC m=+188.233751629" watchObservedRunningTime="2026-05-28 00:14:28.612943072 +0000 UTC m=+188.233975085"
May 28 00:14:28.639593 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:28.639546    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.466548509 podStartE2EDuration="8.639530244s" podCreationTimestamp="2026-05-28 00:14:20 +0000 UTC" firstStartedPulling="2026-05-28 00:14:22.199580494 +0000 UTC m=+181.820612486" lastFinishedPulling="2026-05-28 00:14:27.372562218 +0000 UTC m=+186.993594221" observedRunningTime="2026-05-28 00:14:28.637558714 +0000 UTC m=+188.258590728" watchObservedRunningTime="2026-05-28 00:14:28.639530244 +0000 UTC m=+188.260562257"
May 28 00:14:29.994587 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:29.994552    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-75db84694b-gr46f"]
May 28 00:14:29.998159 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:29.998141    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.005925 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.005897    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 28 00:14:30.008225 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.008199    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75db84694b-gr46f"]
May 28 00:14:30.070919 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.070865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-oauth-serving-cert\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.070919 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.070922    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-console-config\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.071158 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.071012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-service-ca\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.071158 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.071071    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nngfr\" (UniqueName: \"kubernetes.io/projected/70eac1fc-775e-46c2-b532-238eddcd9681-kube-api-access-nngfr\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.071158 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.071108    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-trusted-ca-bundle\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.071307 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.071203    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70eac1fc-775e-46c2-b532-238eddcd9681-console-serving-cert\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.071307 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.071242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70eac1fc-775e-46c2-b532-238eddcd9681-console-oauth-config\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.172595 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.172560    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-service-ca\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.172794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.172613    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nngfr\" (UniqueName: \"kubernetes.io/projected/70eac1fc-775e-46c2-b532-238eddcd9681-kube-api-access-nngfr\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.172794 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.172640    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-trusted-ca-bundle\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.172919 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.172864    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70eac1fc-775e-46c2-b532-238eddcd9681-console-serving-cert\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.172919 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.172901    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70eac1fc-775e-46c2-b532-238eddcd9681-console-oauth-config\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.173018 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.172953    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-oauth-serving-cert\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.173018 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.172988    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-console-config\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.173355 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.173320    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-service-ca\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.173562 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.173542    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-trusted-ca-bundle\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.173658 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.173633    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-oauth-serving-cert\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.173695 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.173646    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-console-config\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.175325 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.175301    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70eac1fc-775e-46c2-b532-238eddcd9681-console-oauth-config\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.175457 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.175438    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70eac1fc-775e-46c2-b532-238eddcd9681-console-serving-cert\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.181375 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.181355    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nngfr\" (UniqueName: \"kubernetes.io/projected/70eac1fc-775e-46c2-b532-238eddcd9681-kube-api-access-nngfr\") pod \"console-75db84694b-gr46f\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") " pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.308612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.308569    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:30.436741 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.436716    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75db84694b-gr46f"]
May 28 00:14:30.439402 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:14:30.439364    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70eac1fc_775e_46c2_b532_238eddcd9681.slice/crio-32764bac2746e7c69e73e4a8030e8c1d062658951690e90a05835aa1dac0733e WatchSource:0}: Error finding container 32764bac2746e7c69e73e4a8030e8c1d062658951690e90a05835aa1dac0733e: Status 404 returned error can't find the container with id 32764bac2746e7c69e73e4a8030e8c1d062658951690e90a05835aa1dac0733e
May 28 00:14:30.600342 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.600252    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75db84694b-gr46f" event={"ID":"70eac1fc-775e-46c2-b532-238eddcd9681","Type":"ContainerStarted","Data":"f15e327a46e104d955464cee9ba67bb510969578692a37f7f1f15ee539097057"}
May 28 00:14:30.600342 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.600288    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75db84694b-gr46f" event={"ID":"70eac1fc-775e-46c2-b532-238eddcd9681","Type":"ContainerStarted","Data":"32764bac2746e7c69e73e4a8030e8c1d062658951690e90a05835aa1dac0733e"}
May 28 00:14:30.618907 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:30.618863    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-75db84694b-gr46f" podStartSLOduration=1.618847434 podStartE2EDuration="1.618847434s" podCreationTimestamp="2026-05-28 00:14:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:14:30.618343255 +0000 UTC m=+190.239375269" watchObservedRunningTime="2026-05-28 00:14:30.618847434 +0000 UTC m=+190.239879447"
May 28 00:14:32.589367 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:32.589337    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-64f86464d4-qvwrw"
May 28 00:14:33.855128 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:33.855096    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:14:40.309200 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:40.309157    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:40.309698 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:40.309213    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:40.315101 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:40.315076    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:40.647402 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:40.647325    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75db84694b-gr46f"
May 28 00:14:44.719032 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:44.718993    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:44.719473 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:44.719080    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:14:45.548177 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:45.548124    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-cbd79fd76-ldsz7" podUID="b6399827-e342-4eed-a849-f7e218bf6bcb" containerName="console" containerID="cri-o://936aef59be9f45617cd6e5a58e6507c2f9dbb6aca400b309d575607e2aaf9a9c" gracePeriod=15
May 28 00:14:45.801576 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:45.801497    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-cbd79fd76-ldsz7_b6399827-e342-4eed-a849-f7e218bf6bcb/console/0.log"
May 28 00:14:45.801961 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:45.801602    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:45.932905 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:45.932863    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-service-ca\") pod \"b6399827-e342-4eed-a849-f7e218bf6bcb\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") "
May 28 00:14:45.933095 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:45.932950    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6399827-e342-4eed-a849-f7e218bf6bcb-console-serving-cert\") pod \"b6399827-e342-4eed-a849-f7e218bf6bcb\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") "
May 28 00:14:45.933095 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:45.932976    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6399827-e342-4eed-a849-f7e218bf6bcb-console-oauth-config\") pod \"b6399827-e342-4eed-a849-f7e218bf6bcb\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") "
May 28 00:14:45.933095 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:45.933006    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-console-config\") pod \"b6399827-e342-4eed-a849-f7e218bf6bcb\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") "
May 28 00:14:45.933095 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:45.933034    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jqdl\" (UniqueName: \"kubernetes.io/projected/b6399827-e342-4eed-a849-f7e218bf6bcb-kube-api-access-4jqdl\") pod \"b6399827-e342-4eed-a849-f7e218bf6bcb\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") "
May 28 00:14:45.933095 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:45.933066    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-oauth-serving-cert\") pod \"b6399827-e342-4eed-a849-f7e218bf6bcb\" (UID: \"b6399827-e342-4eed-a849-f7e218bf6bcb\") "
May 28 00:14:45.933398 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:45.933365    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-service-ca" (OuterVolumeSpecName: "service-ca") pod "b6399827-e342-4eed-a849-f7e218bf6bcb" (UID: "b6399827-e342-4eed-a849-f7e218bf6bcb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:14:45.933458 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:45.933435    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-console-config" (OuterVolumeSpecName: "console-config") pod "b6399827-e342-4eed-a849-f7e218bf6bcb" (UID: "b6399827-e342-4eed-a849-f7e218bf6bcb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:14:45.933567 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:45.933541    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b6399827-e342-4eed-a849-f7e218bf6bcb" (UID: "b6399827-e342-4eed-a849-f7e218bf6bcb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:14:45.935283 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:45.935255    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6399827-e342-4eed-a849-f7e218bf6bcb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b6399827-e342-4eed-a849-f7e218bf6bcb" (UID: "b6399827-e342-4eed-a849-f7e218bf6bcb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:14:45.935405 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:45.935307    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6399827-e342-4eed-a849-f7e218bf6bcb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b6399827-e342-4eed-a849-f7e218bf6bcb" (UID: "b6399827-e342-4eed-a849-f7e218bf6bcb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:14:45.935405 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:45.935313    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6399827-e342-4eed-a849-f7e218bf6bcb-kube-api-access-4jqdl" (OuterVolumeSpecName: "kube-api-access-4jqdl") pod "b6399827-e342-4eed-a849-f7e218bf6bcb" (UID: "b6399827-e342-4eed-a849-f7e218bf6bcb"). InnerVolumeSpecName "kube-api-access-4jqdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:14:46.033858 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.033820    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:14:46.033858 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.033850    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6399827-e342-4eed-a849-f7e218bf6bcb-console-serving-cert\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:14:46.033858 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.033860    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6399827-e342-4eed-a849-f7e218bf6bcb-console-oauth-config\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:14:46.034089 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.033870    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-console-config\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:14:46.034089 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.033879    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jqdl\" (UniqueName: \"kubernetes.io/projected/b6399827-e342-4eed-a849-f7e218bf6bcb-kube-api-access-4jqdl\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:14:46.034089 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.033888    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6399827-e342-4eed-a849-f7e218bf6bcb-oauth-serving-cert\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:14:46.664670 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.664638    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-cbd79fd76-ldsz7_b6399827-e342-4eed-a849-f7e218bf6bcb/console/0.log"
May 28 00:14:46.664845 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.664684    2576 generic.go:358] "Generic (PLEG): container finished" podID="b6399827-e342-4eed-a849-f7e218bf6bcb" containerID="936aef59be9f45617cd6e5a58e6507c2f9dbb6aca400b309d575607e2aaf9a9c" exitCode=2
May 28 00:14:46.664845 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.664751    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbd79fd76-ldsz7"
May 28 00:14:46.664845 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.664773    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbd79fd76-ldsz7" event={"ID":"b6399827-e342-4eed-a849-f7e218bf6bcb","Type":"ContainerDied","Data":"936aef59be9f45617cd6e5a58e6507c2f9dbb6aca400b309d575607e2aaf9a9c"}
May 28 00:14:46.664845 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.664816    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbd79fd76-ldsz7" event={"ID":"b6399827-e342-4eed-a849-f7e218bf6bcb","Type":"ContainerDied","Data":"230342f1065224f5c98e7120cdf23c31ce50f6eb9e3426cfb9932fbf899389dd"}
May 28 00:14:46.664845 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.664833    2576 scope.go:117] "RemoveContainer" containerID="936aef59be9f45617cd6e5a58e6507c2f9dbb6aca400b309d575607e2aaf9a9c"
May 28 00:14:46.679156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.679135    2576 scope.go:117] "RemoveContainer" containerID="936aef59be9f45617cd6e5a58e6507c2f9dbb6aca400b309d575607e2aaf9a9c"
May 28 00:14:46.679460 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:14:46.679437    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936aef59be9f45617cd6e5a58e6507c2f9dbb6aca400b309d575607e2aaf9a9c\": container with ID starting with 936aef59be9f45617cd6e5a58e6507c2f9dbb6aca400b309d575607e2aaf9a9c not found: ID does not exist" containerID="936aef59be9f45617cd6e5a58e6507c2f9dbb6aca400b309d575607e2aaf9a9c"
May 28 00:14:46.679544 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.679467    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936aef59be9f45617cd6e5a58e6507c2f9dbb6aca400b309d575607e2aaf9a9c"} err="failed to get container status \"936aef59be9f45617cd6e5a58e6507c2f9dbb6aca400b309d575607e2aaf9a9c\": rpc error: code = NotFound desc = could not find container \"936aef59be9f45617cd6e5a58e6507c2f9dbb6aca400b309d575607e2aaf9a9c\": container with ID starting with 936aef59be9f45617cd6e5a58e6507c2f9dbb6aca400b309d575607e2aaf9a9c not found: ID does not exist"
May 28 00:14:46.690878 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.690850    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cbd79fd76-ldsz7"]
May 28 00:14:46.692298 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:46.692278    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cbd79fd76-ldsz7"]
May 28 00:14:47.016304 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:47.016233    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6399827-e342-4eed-a849-f7e218bf6bcb" path="/var/lib/kubelet/pods/b6399827-e342-4eed-a849-f7e218bf6bcb/volumes"
May 28 00:14:48.875220 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:48.875165    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5" podUID="0a0045af-7f37-4ae7-89ec-7c66c799c21c" containerName="registry" containerID="cri-o://43b2b6b8c1b8a579f5a40b561118f68e56fb6121d5271dac947ab9f7c880f36f" gracePeriod=30
May 28 00:14:49.119440 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.119416    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:14:49.267023 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.266918    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a0045af-7f37-4ae7-89ec-7c66c799c21c-trusted-ca\") pod \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") "
May 28 00:14:49.267023 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.266991    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a0045af-7f37-4ae7-89ec-7c66c799c21c-installation-pull-secrets\") pod \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") "
May 28 00:14:49.267023 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.267028    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t46r\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-kube-api-access-5t46r\") pod \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") "
May 28 00:14:49.267301 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.267058    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a0045af-7f37-4ae7-89ec-7c66c799c21c-ca-trust-extracted\") pod \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") "
May 28 00:14:49.267301 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.267103    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-certificates\") pod \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") "
May 28 00:14:49.267301 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.267118    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls\") pod \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") "
May 28 00:14:49.267301 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.267158    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a0045af-7f37-4ae7-89ec-7c66c799c21c-image-registry-private-configuration\") pod \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") "
May 28 00:14:49.267301 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.267173    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-bound-sa-token\") pod \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\" (UID: \"0a0045af-7f37-4ae7-89ec-7c66c799c21c\") "
May 28 00:14:49.267564 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.267421    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0045af-7f37-4ae7-89ec-7c66c799c21c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0a0045af-7f37-4ae7-89ec-7c66c799c21c" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:14:49.267618 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.267592    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0a0045af-7f37-4ae7-89ec-7c66c799c21c" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:14:49.269695 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.269663    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0a0045af-7f37-4ae7-89ec-7c66c799c21c" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:14:49.269823 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.269688    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0a0045af-7f37-4ae7-89ec-7c66c799c21c" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:14:49.269888 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.269858    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0045af-7f37-4ae7-89ec-7c66c799c21c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0a0045af-7f37-4ae7-89ec-7c66c799c21c" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:14:49.269941 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.269920    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0045af-7f37-4ae7-89ec-7c66c799c21c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0a0045af-7f37-4ae7-89ec-7c66c799c21c" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:14:49.270059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.270043    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-kube-api-access-5t46r" (OuterVolumeSpecName: "kube-api-access-5t46r") pod "0a0045af-7f37-4ae7-89ec-7c66c799c21c" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c"). InnerVolumeSpecName "kube-api-access-5t46r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:14:49.275639 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.275618    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0045af-7f37-4ae7-89ec-7c66c799c21c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0a0045af-7f37-4ae7-89ec-7c66c799c21c" (UID: "0a0045af-7f37-4ae7-89ec-7c66c799c21c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:14:49.368149 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.368100    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a0045af-7f37-4ae7-89ec-7c66c799c21c-installation-pull-secrets\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:14:49.368149 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.368146    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t46r\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-kube-api-access-5t46r\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:14:49.368149 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.368156    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a0045af-7f37-4ae7-89ec-7c66c799c21c-ca-trust-extracted\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:14:49.368379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.368166    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-certificates\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:14:49.368379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.368176    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-registry-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:14:49.368379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.368187    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a0045af-7f37-4ae7-89ec-7c66c799c21c-image-registry-private-configuration\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:14:49.368379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.368196    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a0045af-7f37-4ae7-89ec-7c66c799c21c-bound-sa-token\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:14:49.368379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.368205    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a0045af-7f37-4ae7-89ec-7c66c799c21c-trusted-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:14:49.675608 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.675572    2576 generic.go:358] "Generic (PLEG): container finished" podID="0a0045af-7f37-4ae7-89ec-7c66c799c21c" containerID="43b2b6b8c1b8a579f5a40b561118f68e56fb6121d5271dac947ab9f7c880f36f" exitCode=0
May 28 00:14:49.675784 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.675631    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5"
May 28 00:14:49.675784 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.675656    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5" event={"ID":"0a0045af-7f37-4ae7-89ec-7c66c799c21c","Type":"ContainerDied","Data":"43b2b6b8c1b8a579f5a40b561118f68e56fb6121d5271dac947ab9f7c880f36f"}
May 28 00:14:49.675784 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.675692    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79b94db5c6-b4nm5" event={"ID":"0a0045af-7f37-4ae7-89ec-7c66c799c21c","Type":"ContainerDied","Data":"0e97ef78d3d31c9478a5663a80fca3f20794f29bea4573c0c97766e7bc5583c1"}
May 28 00:14:49.675784 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.675713    2576 scope.go:117] "RemoveContainer" containerID="43b2b6b8c1b8a579f5a40b561118f68e56fb6121d5271dac947ab9f7c880f36f"
May 28 00:14:49.677269 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.677243    2576 generic.go:358] "Generic (PLEG): container finished" podID="fa182962-797b-4e21-a57f-7225c6bc806a" containerID="e2b51efbd1367a1c36ccc64969fe837d8d1c29fe1a95b7dccbe8352b6ce2d1dc" exitCode=0
May 28 00:14:49.677372 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.677317    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-wfff8" event={"ID":"fa182962-797b-4e21-a57f-7225c6bc806a","Type":"ContainerDied","Data":"e2b51efbd1367a1c36ccc64969fe837d8d1c29fe1a95b7dccbe8352b6ce2d1dc"}
May 28 00:14:49.677631 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.677615    2576 scope.go:117] "RemoveContainer" containerID="e2b51efbd1367a1c36ccc64969fe837d8d1c29fe1a95b7dccbe8352b6ce2d1dc"
May 28 00:14:49.685233 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.685212    2576 scope.go:117] "RemoveContainer" containerID="43b2b6b8c1b8a579f5a40b561118f68e56fb6121d5271dac947ab9f7c880f36f"
May 28 00:14:49.685478 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:14:49.685458    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b2b6b8c1b8a579f5a40b561118f68e56fb6121d5271dac947ab9f7c880f36f\": container with ID starting with 43b2b6b8c1b8a579f5a40b561118f68e56fb6121d5271dac947ab9f7c880f36f not found: ID does not exist" containerID="43b2b6b8c1b8a579f5a40b561118f68e56fb6121d5271dac947ab9f7c880f36f"
May 28 00:14:49.685550 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.685490    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b2b6b8c1b8a579f5a40b561118f68e56fb6121d5271dac947ab9f7c880f36f"} err="failed to get container status \"43b2b6b8c1b8a579f5a40b561118f68e56fb6121d5271dac947ab9f7c880f36f\": rpc error: code = NotFound desc = could not find container \"43b2b6b8c1b8a579f5a40b561118f68e56fb6121d5271dac947ab9f7c880f36f\": container with ID starting with 43b2b6b8c1b8a579f5a40b561118f68e56fb6121d5271dac947ab9f7c880f36f not found: ID does not exist"
May 28 00:14:49.714962 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.714932    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79b94db5c6-b4nm5"]
May 28 00:14:49.721947 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:49.721923    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79b94db5c6-b4nm5"]
May 28 00:14:50.682691 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:50.682651    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-wfff8" event={"ID":"fa182962-797b-4e21-a57f-7225c6bc806a","Type":"ContainerStarted","Data":"63ceefc87cf31b8e85e32c96355722fcbf260387c639277293d69331a9651318"}
May 28 00:14:51.021024 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:51.020947    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a0045af-7f37-4ae7-89ec-7c66c799c21c" path="/var/lib/kubelet/pods/0a0045af-7f37-4ae7-89ec-7c66c799c21c/volumes"
May 28 00:14:59.714078 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:59.714041    2576 generic.go:358] "Generic (PLEG): container finished" podID="43cdd4d1-8b9f-4b27-8280-5b668415be73" containerID="318b68c60e3919c0a25ea5c581fa52e3e9a3f6af066601cfae781f3c7e26457a" exitCode=0
May 28 00:14:59.714421 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:59.714090    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb" event={"ID":"43cdd4d1-8b9f-4b27-8280-5b668415be73","Type":"ContainerDied","Data":"318b68c60e3919c0a25ea5c581fa52e3e9a3f6af066601cfae781f3c7e26457a"}
May 28 00:14:59.714421 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:14:59.714406    2576 scope.go:117] "RemoveContainer" containerID="318b68c60e3919c0a25ea5c581fa52e3e9a3f6af066601cfae781f3c7e26457a"
May 28 00:15:00.719276 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:00.719243    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-j9fwb" event={"ID":"43cdd4d1-8b9f-4b27-8280-5b668415be73","Type":"ContainerStarted","Data":"5727da8f58cbcf3ad6b9672bd68c0e4d8006d747d5d5b8705703e7ce21f49baa"}
May 28 00:15:04.725571 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:04.725539    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:15:04.730319 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:04.730289    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-784984c5dc-64l2b"
May 28 00:15:32.751640 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:32.751598    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs\") pod \"network-metrics-daemon-6b84s\" (UID: \"653f150e-1c4b-4dc5-a088-6ab0ad385350\") " pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:15:32.753885 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:32.753856    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653f150e-1c4b-4dc5-a088-6ab0ad385350-metrics-certs\") pod \"network-metrics-daemon-6b84s\" (UID: \"653f150e-1c4b-4dc5-a088-6ab0ad385350\") " pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:15:33.015450 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:33.015362    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fp9tk\""
May 28 00:15:33.023204 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:33.023182    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b84s"
May 28 00:15:33.150128 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:33.150105    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6b84s"]
May 28 00:15:33.152421 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:15:33.152394    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653f150e_1c4b_4dc5_a088_6ab0ad385350.slice/crio-0628db8524c7b3aca11990cf9e0fec35a7bead9002372c7880c8542663927c9c WatchSource:0}: Error finding container 0628db8524c7b3aca11990cf9e0fec35a7bead9002372c7880c8542663927c9c: Status 404 returned error can't find the container with id 0628db8524c7b3aca11990cf9e0fec35a7bead9002372c7880c8542663927c9c
May 28 00:15:33.831441 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:33.831403    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6b84s" event={"ID":"653f150e-1c4b-4dc5-a088-6ab0ad385350","Type":"ContainerStarted","Data":"0628db8524c7b3aca11990cf9e0fec35a7bead9002372c7880c8542663927c9c"}
May 28 00:15:34.836401 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:34.836365    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6b84s" event={"ID":"653f150e-1c4b-4dc5-a088-6ab0ad385350","Type":"ContainerStarted","Data":"3a2f7e74fdb33673c3e2767d28364e1afc0491887f2d03fc65c4cbcc72f5d637"}
May 28 00:15:34.836401 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:34.836402    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6b84s" event={"ID":"653f150e-1c4b-4dc5-a088-6ab0ad385350","Type":"ContainerStarted","Data":"d2725ffd4a7dbf48c3d1fb8b3e7d7818993bb697bbfc1dd82414c469bca58b40"}
May 28 00:15:34.853161 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:34.852484    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6b84s" podStartSLOduration=252.72201514 podStartE2EDuration="4m13.852465494s" podCreationTimestamp="2026-05-28 00:11:21 +0000 UTC" firstStartedPulling="2026-05-28 00:15:33.154275498 +0000 UTC m=+252.775307506" lastFinishedPulling="2026-05-28 00:15:34.284725869 +0000 UTC m=+253.905757860" observedRunningTime="2026-05-28 00:15:34.85244915 +0000 UTC m=+254.473481164" watchObservedRunningTime="2026-05-28 00:15:34.852465494 +0000 UTC m=+254.473497508"
May 28 00:15:57.247878 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.247841    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-696888d86-kp2s8"]
May 28 00:15:57.248632 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.248371    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6399827-e342-4eed-a849-f7e218bf6bcb" containerName="console"
May 28 00:15:57.248632 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.248390    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6399827-e342-4eed-a849-f7e218bf6bcb" containerName="console"
May 28 00:15:57.248632 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.248407    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a0045af-7f37-4ae7-89ec-7c66c799c21c" containerName="registry"
May 28 00:15:57.248632 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.248416    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a0045af-7f37-4ae7-89ec-7c66c799c21c" containerName="registry"
May 28 00:15:57.248632 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.248548    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="b6399827-e342-4eed-a849-f7e218bf6bcb" containerName="console"
May 28 00:15:57.248632 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.248564    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="0a0045af-7f37-4ae7-89ec-7c66c799c21c" containerName="registry"
May 28 00:15:57.251911 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.251888    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.263050 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.262951    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-696888d86-kp2s8"]
May 28 00:15:57.266654 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.266622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-service-ca\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.266802 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.266670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-console-config\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.266802 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.266760    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dr62\" (UniqueName: \"kubernetes.io/projected/ced61fe7-8b28-4567-a324-40a9479c5359-kube-api-access-7dr62\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.266919 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.266807    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-trusted-ca-bundle\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.266919 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.266869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ced61fe7-8b28-4567-a324-40a9479c5359-console-oauth-config\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.266919 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.266901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-oauth-serving-cert\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.267087 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.266945    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ced61fe7-8b28-4567-a324-40a9479c5359-console-serving-cert\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.367840 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.367805    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ced61fe7-8b28-4567-a324-40a9479c5359-console-serving-cert\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.368025 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.367864    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-service-ca\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.368025 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.367979    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-console-config\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.368154 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.368034    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7dr62\" (UniqueName: \"kubernetes.io/projected/ced61fe7-8b28-4567-a324-40a9479c5359-kube-api-access-7dr62\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.368154 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.368072    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-trusted-ca-bundle\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.368154 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.368109    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ced61fe7-8b28-4567-a324-40a9479c5359-console-oauth-config\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.368154 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.368142    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-oauth-serving-cert\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.368643 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.368617    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-service-ca\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.368747 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.368692    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-console-config\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.368834 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.368811    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-oauth-serving-cert\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.368935 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.368917    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-trusted-ca-bundle\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.370494 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.370467    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ced61fe7-8b28-4567-a324-40a9479c5359-console-serving-cert\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.370494 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.370482    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ced61fe7-8b28-4567-a324-40a9479c5359-console-oauth-config\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.379833 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.379804    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dr62\" (UniqueName: \"kubernetes.io/projected/ced61fe7-8b28-4567-a324-40a9479c5359-kube-api-access-7dr62\") pod \"console-696888d86-kp2s8\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") " pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.561788 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.561745    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696888d86-kp2s8"
May 28 00:15:57.696211 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.696181    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-696888d86-kp2s8"]
May 28 00:15:57.698911 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:15:57.698885    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced61fe7_8b28_4567_a324_40a9479c5359.slice/crio-bc33684ed1113db2bb948239396e2a956ce06b17ab06f814159daa1a63a83edb WatchSource:0}: Error finding container bc33684ed1113db2bb948239396e2a956ce06b17ab06f814159daa1a63a83edb: Status 404 returned error can't find the container with id bc33684ed1113db2bb948239396e2a956ce06b17ab06f814159daa1a63a83edb
May 28 00:15:57.915422 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.915323    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696888d86-kp2s8" event={"ID":"ced61fe7-8b28-4567-a324-40a9479c5359","Type":"ContainerStarted","Data":"c5b2963edb1f9a1bc766b2ec0ce33c4c94604a29574bdcda56f54e379744fea1"}
May 28 00:15:57.915422 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.915367    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696888d86-kp2s8" event={"ID":"ced61fe7-8b28-4567-a324-40a9479c5359","Type":"ContainerStarted","Data":"bc33684ed1113db2bb948239396e2a956ce06b17ab06f814159daa1a63a83edb"}
May 28 00:15:57.935152 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:15:57.935100    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-696888d86-kp2s8" podStartSLOduration=0.935084859 podStartE2EDuration="935.084859ms" podCreationTimestamp="2026-05-28 00:15:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:15:57.933626349 +0000 UTC m=+277.554658362" watchObservedRunningTime="2026-05-28 00:15:57.935084859 +0000 UTC m=+277.556116870"
May 28 00:16:07.562688 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:07.562651    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-696888d86-kp2s8"
May 28 00:16:07.563102 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:07.562741    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-696888d86-kp2s8"
May 28 00:16:07.568273 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:07.568246    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-696888d86-kp2s8"
May 28 00:16:07.954735 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:07.954655    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-696888d86-kp2s8"
May 28 00:16:08.007101 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:08.007069    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75db84694b-gr46f"]
May 28 00:16:20.899055 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:20.899026    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:16:20.899532 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:20.899499    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:16:20.908671 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:20.908640    2576 kubelet.go:1628] "Image garbage collection succeeded"
May 28 00:16:33.027957 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.027898    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-75db84694b-gr46f" podUID="70eac1fc-775e-46c2-b532-238eddcd9681" containerName="console" containerID="cri-o://f15e327a46e104d955464cee9ba67bb510969578692a37f7f1f15ee539097057" gracePeriod=15
May 28 00:16:33.278035 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.277973    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-75db84694b-gr46f_70eac1fc-775e-46c2-b532-238eddcd9681/console/0.log"
May 28 00:16:33.278191 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.278039    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75db84694b-gr46f"
May 28 00:16:33.392426 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.392393    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-service-ca\") pod \"70eac1fc-775e-46c2-b532-238eddcd9681\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") "
May 28 00:16:33.392647 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.392470    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nngfr\" (UniqueName: \"kubernetes.io/projected/70eac1fc-775e-46c2-b532-238eddcd9681-kube-api-access-nngfr\") pod \"70eac1fc-775e-46c2-b532-238eddcd9681\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") "
May 28 00:16:33.392647 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.392539    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70eac1fc-775e-46c2-b532-238eddcd9681-console-serving-cert\") pod \"70eac1fc-775e-46c2-b532-238eddcd9681\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") "
May 28 00:16:33.392771 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.392651    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-trusted-ca-bundle\") pod \"70eac1fc-775e-46c2-b532-238eddcd9681\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") "
May 28 00:16:33.392771 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.392732    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70eac1fc-775e-46c2-b532-238eddcd9681-console-oauth-config\") pod \"70eac1fc-775e-46c2-b532-238eddcd9681\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") "
May 28 00:16:33.392771 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.392760    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-oauth-serving-cert\") pod \"70eac1fc-775e-46c2-b532-238eddcd9681\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") "
May 28 00:16:33.392922 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.392820    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-console-config\") pod \"70eac1fc-775e-46c2-b532-238eddcd9681\" (UID: \"70eac1fc-775e-46c2-b532-238eddcd9681\") "
May 28 00:16:33.392922 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.392823    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-service-ca" (OuterVolumeSpecName: "service-ca") pod "70eac1fc-775e-46c2-b532-238eddcd9681" (UID: "70eac1fc-775e-46c2-b532-238eddcd9681"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:16:33.393112 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.393082    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:16:33.393239 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.393102    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "70eac1fc-775e-46c2-b532-238eddcd9681" (UID: "70eac1fc-775e-46c2-b532-238eddcd9681"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:16:33.393317 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.393291    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-console-config" (OuterVolumeSpecName: "console-config") pod "70eac1fc-775e-46c2-b532-238eddcd9681" (UID: "70eac1fc-775e-46c2-b532-238eddcd9681"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:16:33.393436 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.393338    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "70eac1fc-775e-46c2-b532-238eddcd9681" (UID: "70eac1fc-775e-46c2-b532-238eddcd9681"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:16:33.394922 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.394894    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70eac1fc-775e-46c2-b532-238eddcd9681-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "70eac1fc-775e-46c2-b532-238eddcd9681" (UID: "70eac1fc-775e-46c2-b532-238eddcd9681"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:16:33.395185 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.395163    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70eac1fc-775e-46c2-b532-238eddcd9681-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "70eac1fc-775e-46c2-b532-238eddcd9681" (UID: "70eac1fc-775e-46c2-b532-238eddcd9681"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:16:33.395260 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.395181    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70eac1fc-775e-46c2-b532-238eddcd9681-kube-api-access-nngfr" (OuterVolumeSpecName: "kube-api-access-nngfr") pod "70eac1fc-775e-46c2-b532-238eddcd9681" (UID: "70eac1fc-775e-46c2-b532-238eddcd9681"). InnerVolumeSpecName "kube-api-access-nngfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:16:33.494322 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.494282    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70eac1fc-775e-46c2-b532-238eddcd9681-console-oauth-config\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:16:33.494322 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.494315    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-oauth-serving-cert\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:16:33.494322 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.494329    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-console-config\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:16:33.494590 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.494342    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nngfr\" (UniqueName: \"kubernetes.io/projected/70eac1fc-775e-46c2-b532-238eddcd9681-kube-api-access-nngfr\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:16:33.494590 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.494354    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70eac1fc-775e-46c2-b532-238eddcd9681-console-serving-cert\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:16:33.494590 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:33.494368    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70eac1fc-775e-46c2-b532-238eddcd9681-trusted-ca-bundle\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:16:34.039699 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:34.039671    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-75db84694b-gr46f_70eac1fc-775e-46c2-b532-238eddcd9681/console/0.log"
May 28 00:16:34.040187 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:34.039711    2576 generic.go:358] "Generic (PLEG): container finished" podID="70eac1fc-775e-46c2-b532-238eddcd9681" containerID="f15e327a46e104d955464cee9ba67bb510969578692a37f7f1f15ee539097057" exitCode=2
May 28 00:16:34.040187 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:34.039803    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75db84694b-gr46f"
May 28 00:16:34.040187 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:34.039810    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75db84694b-gr46f" event={"ID":"70eac1fc-775e-46c2-b532-238eddcd9681","Type":"ContainerDied","Data":"f15e327a46e104d955464cee9ba67bb510969578692a37f7f1f15ee539097057"}
May 28 00:16:34.040187 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:34.039856    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75db84694b-gr46f" event={"ID":"70eac1fc-775e-46c2-b532-238eddcd9681","Type":"ContainerDied","Data":"32764bac2746e7c69e73e4a8030e8c1d062658951690e90a05835aa1dac0733e"}
May 28 00:16:34.040187 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:34.039874    2576 scope.go:117] "RemoveContainer" containerID="f15e327a46e104d955464cee9ba67bb510969578692a37f7f1f15ee539097057"
May 28 00:16:34.049036 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:34.049014    2576 scope.go:117] "RemoveContainer" containerID="f15e327a46e104d955464cee9ba67bb510969578692a37f7f1f15ee539097057"
May 28 00:16:34.049315 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:16:34.049297    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f15e327a46e104d955464cee9ba67bb510969578692a37f7f1f15ee539097057\": container with ID starting with f15e327a46e104d955464cee9ba67bb510969578692a37f7f1f15ee539097057 not found: ID does not exist" containerID="f15e327a46e104d955464cee9ba67bb510969578692a37f7f1f15ee539097057"
May 28 00:16:34.049356 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:34.049324    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15e327a46e104d955464cee9ba67bb510969578692a37f7f1f15ee539097057"} err="failed to get container status \"f15e327a46e104d955464cee9ba67bb510969578692a37f7f1f15ee539097057\": rpc error: code = NotFound desc = could not find container \"f15e327a46e104d955464cee9ba67bb510969578692a37f7f1f15ee539097057\": container with ID starting with f15e327a46e104d955464cee9ba67bb510969578692a37f7f1f15ee539097057 not found: ID does not exist"
May 28 00:16:34.061966 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:34.061939    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75db84694b-gr46f"]
May 28 00:16:34.066488 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:34.066462    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75db84694b-gr46f"]
May 28 00:16:35.015872 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:16:35.015839    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70eac1fc-775e-46c2-b532-238eddcd9681" path="/var/lib/kubelet/pods/70eac1fc-775e-46c2-b532-238eddcd9681/volumes"
May 28 00:18:41.949429 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:41.949308    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"]
May 28 00:18:41.950138 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:41.949840    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70eac1fc-775e-46c2-b532-238eddcd9681" containerName="console"
May 28 00:18:41.950138 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:41.949860    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="70eac1fc-775e-46c2-b532-238eddcd9681" containerName="console"
May 28 00:18:41.950138 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:41.949952    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="70eac1fc-775e-46c2-b532-238eddcd9681" containerName="console"
May 28 00:18:41.953446 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:41.953423    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:41.956108 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:41.956077    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 00:18:41.956258 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:41.956164    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-p7qn7\""
May 28 00:18:41.956258 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:41.956174    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 28 00:18:41.956258 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:41.956228    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 28 00:18:41.956414 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:41.956328    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 28 00:18:41.956557 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:41.956540    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 00:18:41.956682 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:41.956557    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 28 00:18:41.956756 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:41.956634    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 28 00:18:41.965331 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:41.965307    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"]
May 28 00:18:42.012633 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.012590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-offline-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.012633 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.012637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-registry-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.012901 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.012665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-ui-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.012901 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.012815    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-online-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.012901 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.012849    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-feast-data\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.013026 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.012905    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whxgt\" (UniqueName: \"kubernetes.io/projected/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-kube-api-access-whxgt\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.113880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.113840    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-offline-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.114086 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.113911    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-registry-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.114086 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.113941    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-ui-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.114086 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.113993    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-online-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.114086 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.114025    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-feast-data\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.114309 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.114110    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-whxgt\" (UniqueName: \"kubernetes.io/projected/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-kube-api-access-whxgt\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.114570 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.114539    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-feast-data\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.116571 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.116549    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-offline-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.116684 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.116601    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-online-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.116684 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.116617    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-ui-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.116767 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.116682    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-registry-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.124301 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.124277    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whxgt\" (UniqueName: \"kubernetes.io/projected/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-kube-api-access-whxgt\") pod \"feast-simple-feast-setup-7cf6cbc6d6-77j4g\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.275673 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.275585    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:18:42.416106 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.416061    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"]
May 28 00:18:42.419677 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:18:42.419633    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod662869fd_dbb9_48e2_8b54_d2bdeddfe6b2.slice/crio-4e9155e683eb0fa9a65a574b746a69df402eb8cf597e8aba5e78f93dfda14b8d WatchSource:0}: Error finding container 4e9155e683eb0fa9a65a574b746a69df402eb8cf597e8aba5e78f93dfda14b8d: Status 404 returned error can't find the container with id 4e9155e683eb0fa9a65a574b746a69df402eb8cf597e8aba5e78f93dfda14b8d
May 28 00:18:42.421423 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.421404    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 00:18:42.440159 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:18:42.440124    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" event={"ID":"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2","Type":"ContainerStarted","Data":"4e9155e683eb0fa9a65a574b746a69df402eb8cf597e8aba5e78f93dfda14b8d"}
May 28 00:19:15.567268 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:15.567234    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" event={"ID":"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2","Type":"ContainerStarted","Data":"94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28"}
May 28 00:19:20.584429 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:20.584394    2576 generic.go:358] "Generic (PLEG): container finished" podID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerID="94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28" exitCode=0
May 28 00:19:20.584825 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:20.584465    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" event={"ID":"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2","Type":"ContainerDied","Data":"94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28"}
May 28 00:19:21.589172 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:21.589140    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" event={"ID":"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2","Type":"ContainerStarted","Data":"b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb"}
May 28 00:19:26.608101 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:26.608067    2576 generic.go:358] "Generic (PLEG): container finished" podID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerID="b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb" exitCode=0
May 28 00:19:26.608478 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:26.608153    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" event={"ID":"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2","Type":"ContainerDied","Data":"b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb"}
May 28 00:19:26.623281 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:26.623256    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 28 00:19:27.616120 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:27.616073    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" event={"ID":"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2","Type":"ContainerStarted","Data":"d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba"}
May 28 00:19:27.616749 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:27.616141    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" event={"ID":"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2","Type":"ContainerStarted","Data":"79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184"}
May 28 00:19:27.616749 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:27.616160    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" event={"ID":"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2","Type":"ContainerStarted","Data":"f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf"}
May 28 00:19:27.616749 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:27.616174    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" event={"ID":"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2","Type":"ContainerStarted","Data":"50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774"}
May 28 00:19:27.642910 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:27.642845    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" podStartSLOduration=13.586642797 podStartE2EDuration="46.642823281s" podCreationTimestamp="2026-05-28 00:18:41 +0000 UTC" firstStartedPulling="2026-05-28 00:18:42.42155253 +0000 UTC m=+442.042584522" lastFinishedPulling="2026-05-28 00:19:15.477733016 +0000 UTC m=+475.098765006" observedRunningTime="2026-05-28 00:19:27.640658196 +0000 UTC m=+487.261690211" watchObservedRunningTime="2026-05-28 00:19:27.642823281 +0000 UTC m=+487.263855296"
May 28 00:19:30.276499 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:30.276395    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:30.276499 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:30.276443    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:30.276499 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:30.276457    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:30.276499 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:30.276472    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:30.279232 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:30.279107    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused"
May 28 00:19:30.279232 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:30.279131    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.29:8816: connect: connection refused"
May 28 00:19:30.279232 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:30.279160    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused"
May 28 00:19:30.279232 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:30.279175    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.29:6571: connect: connection refused"
May 28 00:19:32.276427 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:32.276376    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:32.276939 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:32.276458    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:32.276939 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:32.276484    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:32.276939 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:32.276496    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:33.276839 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:33.276808    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:33.277283 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:33.277029    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:33.277283 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:33.277229    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:33.281330 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:33.281307    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:33.641645 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:33.641617    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:33.641813 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:33.641681    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:33.641813 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:33.641714    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:33.644115 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:33.644090    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:19:52.446198 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:19:52.446159    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-696888d86-kp2s8"]
May 28 00:20:17.467573 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.467427    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-696888d86-kp2s8" podUID="ced61fe7-8b28-4567-a324-40a9479c5359" containerName="console" containerID="cri-o://c5b2963edb1f9a1bc766b2ec0ce33c4c94604a29574bdcda56f54e379744fea1" gracePeriod=15
May 28 00:20:17.720336 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.720278    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-696888d86-kp2s8_ced61fe7-8b28-4567-a324-40a9479c5359/console/0.log"
May 28 00:20:17.720450 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.720340    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696888d86-kp2s8"
May 28 00:20:17.778001 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.777968    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ced61fe7-8b28-4567-a324-40a9479c5359-console-oauth-config\") pod \"ced61fe7-8b28-4567-a324-40a9479c5359\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") "
May 28 00:20:17.778176 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.778010    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-trusted-ca-bundle\") pod \"ced61fe7-8b28-4567-a324-40a9479c5359\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") "
May 28 00:20:17.778176 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.778072    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-oauth-serving-cert\") pod \"ced61fe7-8b28-4567-a324-40a9479c5359\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") "
May 28 00:20:17.778176 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.778103    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dr62\" (UniqueName: \"kubernetes.io/projected/ced61fe7-8b28-4567-a324-40a9479c5359-kube-api-access-7dr62\") pod \"ced61fe7-8b28-4567-a324-40a9479c5359\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") "
May 28 00:20:17.778176 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.778138    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-console-config\") pod \"ced61fe7-8b28-4567-a324-40a9479c5359\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") "
May 28 00:20:17.778176 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.778172    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-service-ca\") pod \"ced61fe7-8b28-4567-a324-40a9479c5359\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") "
May 28 00:20:17.778443 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.778207    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ced61fe7-8b28-4567-a324-40a9479c5359-console-serving-cert\") pod \"ced61fe7-8b28-4567-a324-40a9479c5359\" (UID: \"ced61fe7-8b28-4567-a324-40a9479c5359\") "
May 28 00:20:17.778591 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.778563    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ced61fe7-8b28-4567-a324-40a9479c5359" (UID: "ced61fe7-8b28-4567-a324-40a9479c5359"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:20:17.778686 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.778592    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ced61fe7-8b28-4567-a324-40a9479c5359" (UID: "ced61fe7-8b28-4567-a324-40a9479c5359"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:20:17.778686 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.778613    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-console-config" (OuterVolumeSpecName: "console-config") pod "ced61fe7-8b28-4567-a324-40a9479c5359" (UID: "ced61fe7-8b28-4567-a324-40a9479c5359"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:20:17.778686 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.778616    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-service-ca" (OuterVolumeSpecName: "service-ca") pod "ced61fe7-8b28-4567-a324-40a9479c5359" (UID: "ced61fe7-8b28-4567-a324-40a9479c5359"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:20:17.780228 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.780203    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced61fe7-8b28-4567-a324-40a9479c5359-kube-api-access-7dr62" (OuterVolumeSpecName: "kube-api-access-7dr62") pod "ced61fe7-8b28-4567-a324-40a9479c5359" (UID: "ced61fe7-8b28-4567-a324-40a9479c5359"). InnerVolumeSpecName "kube-api-access-7dr62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:20:17.780324 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.780242    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced61fe7-8b28-4567-a324-40a9479c5359-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ced61fe7-8b28-4567-a324-40a9479c5359" (UID: "ced61fe7-8b28-4567-a324-40a9479c5359"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:20:17.780365 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.780332    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced61fe7-8b28-4567-a324-40a9479c5359-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ced61fe7-8b28-4567-a324-40a9479c5359" (UID: "ced61fe7-8b28-4567-a324-40a9479c5359"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:20:17.801787 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.801762    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-696888d86-kp2s8_ced61fe7-8b28-4567-a324-40a9479c5359/console/0.log"
May 28 00:20:17.801891 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.801803    2576 generic.go:358] "Generic (PLEG): container finished" podID="ced61fe7-8b28-4567-a324-40a9479c5359" containerID="c5b2963edb1f9a1bc766b2ec0ce33c4c94604a29574bdcda56f54e379744fea1" exitCode=2
May 28 00:20:17.801891 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.801856    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696888d86-kp2s8" event={"ID":"ced61fe7-8b28-4567-a324-40a9479c5359","Type":"ContainerDied","Data":"c5b2963edb1f9a1bc766b2ec0ce33c4c94604a29574bdcda56f54e379744fea1"}
May 28 00:20:17.801891 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.801876    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-696888d86-kp2s8"
May 28 00:20:17.801891 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.801887    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-696888d86-kp2s8" event={"ID":"ced61fe7-8b28-4567-a324-40a9479c5359","Type":"ContainerDied","Data":"bc33684ed1113db2bb948239396e2a956ce06b17ab06f814159daa1a63a83edb"}
May 28 00:20:17.802025 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.801908    2576 scope.go:117] "RemoveContainer" containerID="c5b2963edb1f9a1bc766b2ec0ce33c4c94604a29574bdcda56f54e379744fea1"
May 28 00:20:17.810972 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.810954    2576 scope.go:117] "RemoveContainer" containerID="c5b2963edb1f9a1bc766b2ec0ce33c4c94604a29574bdcda56f54e379744fea1"
May 28 00:20:17.811215 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:20:17.811196    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b2963edb1f9a1bc766b2ec0ce33c4c94604a29574bdcda56f54e379744fea1\": container with ID starting with c5b2963edb1f9a1bc766b2ec0ce33c4c94604a29574bdcda56f54e379744fea1 not found: ID does not exist" containerID="c5b2963edb1f9a1bc766b2ec0ce33c4c94604a29574bdcda56f54e379744fea1"
May 28 00:20:17.811282 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.811222    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b2963edb1f9a1bc766b2ec0ce33c4c94604a29574bdcda56f54e379744fea1"} err="failed to get container status \"c5b2963edb1f9a1bc766b2ec0ce33c4c94604a29574bdcda56f54e379744fea1\": rpc error: code = NotFound desc = could not find container \"c5b2963edb1f9a1bc766b2ec0ce33c4c94604a29574bdcda56f54e379744fea1\": container with ID starting with c5b2963edb1f9a1bc766b2ec0ce33c4c94604a29574bdcda56f54e379744fea1 not found: ID does not exist"
May 28 00:20:17.828757 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.828730    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-696888d86-kp2s8"]
May 28 00:20:17.832918 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.832897    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-696888d86-kp2s8"]
May 28 00:20:17.879563 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.879535    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-oauth-serving-cert\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:20:17.879563 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.879561    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dr62\" (UniqueName: \"kubernetes.io/projected/ced61fe7-8b28-4567-a324-40a9479c5359-kube-api-access-7dr62\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:20:17.879714 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.879575    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-console-config\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:20:17.879714 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.879587    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-service-ca\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:20:17.879714 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.879595    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ced61fe7-8b28-4567-a324-40a9479c5359-console-serving-cert\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:20:17.879714 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.879603    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ced61fe7-8b28-4567-a324-40a9479c5359-console-oauth-config\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:20:17.879714 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:17.879611    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ced61fe7-8b28-4567-a324-40a9479c5359-trusted-ca-bundle\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:20:19.016430 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:19.016397    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ced61fe7-8b28-4567-a324-40a9479c5359" path="/var/lib/kubelet/pods/ced61fe7-8b28-4567-a324-40a9479c5359/volumes"
May 28 00:20:35.229242 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.229209    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"]
May 28 00:20:35.229858 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.229545    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="registry" containerID="cri-o://50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774" gracePeriod=30
May 28 00:20:35.229858 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.229639    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="ui" containerID="cri-o://d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba" gracePeriod=30
May 28 00:20:35.229858 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.229670    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="offline" containerID="cri-o://79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184" gracePeriod=30
May 28 00:20:35.229858 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.229735    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="online" containerID="cri-o://f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf" gracePeriod=30
May 28 00:20:35.500858 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.500771    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"]
May 28 00:20:35.501601 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.501575    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced61fe7-8b28-4567-a324-40a9479c5359" containerName="console"
May 28 00:20:35.501601 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.501598    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced61fe7-8b28-4567-a324-40a9479c5359" containerName="console"
May 28 00:20:35.501797 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.501681    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="ced61fe7-8b28-4567-a324-40a9479c5359" containerName="console"
May 28 00:20:35.507361 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.507340    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.510099 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.510076    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-sgsft\""
May 28 00:20:35.519301 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.519269    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"]
May 28 00:20:35.638862 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.638805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-registry-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.639054 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.638936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-offline-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.639054 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.639004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-ui-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.639174 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.639144    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc38c66-9b71-4d63-a1be-b83d30a52103-feast-data\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.639225 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.639171    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-online-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.639225 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.639217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2pn6\" (UniqueName: \"kubernetes.io/projected/7bc38c66-9b71-4d63-a1be-b83d30a52103-kube-api-access-l2pn6\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.740497 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.740454    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-registry-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.740710 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.740535    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-offline-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.740710 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.740562    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-ui-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.740710 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:20:35.740635    2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 28 00:20:35.740710 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.740658    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc38c66-9b71-4d63-a1be-b83d30a52103-feast-data\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.740710 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.740689    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-online-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.740962 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:20:35.740717    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-registry-tls podName:7bc38c66-9b71-4d63-a1be-b83d30a52103 nodeName:}" failed. No retries permitted until 2026-05-28 00:20:36.240696109 +0000 UTC m=+555.861728101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-registry-tls") pod "feast-simple-feast-setup-7cf6cbc6d6-6pkd5" (UID: "7bc38c66-9b71-4d63-a1be-b83d30a52103") : secret "feast-simple-feast-setup-registry-tls" not found
May 28 00:20:35.740962 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.740765    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l2pn6\" (UniqueName: \"kubernetes.io/projected/7bc38c66-9b71-4d63-a1be-b83d30a52103-kube-api-access-l2pn6\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.741229 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:20:35.741202    2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 28 00:20:35.741332 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:20:35.741255    2576 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 28 00:20:35.741332 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:20:35.741268    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-offline-tls podName:7bc38c66-9b71-4d63-a1be-b83d30a52103 nodeName:}" failed. No retries permitted until 2026-05-28 00:20:36.241250971 +0000 UTC m=+555.862282977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-offline-tls") pod "feast-simple-feast-setup-7cf6cbc6d6-6pkd5" (UID: "7bc38c66-9b71-4d63-a1be-b83d30a52103") : secret "feast-simple-feast-setup-offline-tls" not found
May 28 00:20:35.741332 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:20:35.741308    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-ui-tls podName:7bc38c66-9b71-4d63-a1be-b83d30a52103 nodeName:}" failed. No retries permitted until 2026-05-28 00:20:36.241292398 +0000 UTC m=+555.862324389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-ui-tls") pod "feast-simple-feast-setup-7cf6cbc6d6-6pkd5" (UID: "7bc38c66-9b71-4d63-a1be-b83d30a52103") : secret "feast-simple-feast-setup-ui-tls" not found
May 28 00:20:35.741629 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.741606    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc38c66-9b71-4d63-a1be-b83d30a52103-feast-data\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.743981 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.743958    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-online-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.750615 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.750585    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2pn6\" (UniqueName: \"kubernetes.io/projected/7bc38c66-9b71-4d63-a1be-b83d30a52103-kube-api-access-l2pn6\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:35.862841 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.862807    2576 generic.go:358] "Generic (PLEG): container finished" podID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerID="d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba" exitCode=0
May 28 00:20:35.862992 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:35.862880    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" event={"ID":"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2","Type":"ContainerDied","Data":"d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba"}
May 28 00:20:36.245381 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:36.245341    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-registry-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:36.245381 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:36.245390    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-offline-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:36.245821 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:36.245540    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-ui-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:36.247897 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:36.247869    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-ui-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:36.247998 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:36.247908    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-offline-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:36.247998 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:36.247919    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-registry-tls\") pod \"feast-simple-feast-setup-7cf6cbc6d6-6pkd5\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") " pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:36.424012 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:36.423962    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:36.554070 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:36.554035    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"]
May 28 00:20:36.557075 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:20:36.557040    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bc38c66_9b71_4d63_a1be_b83d30a52103.slice/crio-57540ae1bb2a4f4e319a0dce1726b64013d8614e987e29cbca4f036de393f72f WatchSource:0}: Error finding container 57540ae1bb2a4f4e319a0dce1726b64013d8614e987e29cbca4f036de393f72f: Status 404 returned error can't find the container with id 57540ae1bb2a4f4e319a0dce1726b64013d8614e987e29cbca4f036de393f72f
May 28 00:20:36.872013 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:36.871977    2576 generic.go:358] "Generic (PLEG): container finished" podID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerID="f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf" exitCode=0
May 28 00:20:36.872206 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:36.872114    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" event={"ID":"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2","Type":"ContainerDied","Data":"f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf"}
May 28 00:20:36.877773 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:36.876908    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" event={"ID":"7bc38c66-9b71-4d63-a1be-b83d30a52103","Type":"ContainerStarted","Data":"b5781521310bd49652eac86956b647d91c4d224d364715446cfdd6bcfdd9ad43"}
May 28 00:20:36.877773 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:36.876947    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" event={"ID":"7bc38c66-9b71-4d63-a1be-b83d30a52103","Type":"ContainerStarted","Data":"57540ae1bb2a4f4e319a0dce1726b64013d8614e987e29cbca4f036de393f72f"}
May 28 00:20:41.897075 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:41.897039    2576 generic.go:358] "Generic (PLEG): container finished" podID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerID="b5781521310bd49652eac86956b647d91c4d224d364715446cfdd6bcfdd9ad43" exitCode=0
May 28 00:20:41.897568 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:41.897096    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" event={"ID":"7bc38c66-9b71-4d63-a1be-b83d30a52103","Type":"ContainerDied","Data":"b5781521310bd49652eac86956b647d91c4d224d364715446cfdd6bcfdd9ad43"}
May 28 00:20:42.903067 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:42.903029    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" event={"ID":"7bc38c66-9b71-4d63-a1be-b83d30a52103","Type":"ContainerStarted","Data":"8efb37107b35aea1f1ff146457b13842ce52855d1771550df9fd2276c619f8aa"}
May 28 00:20:43.641307 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:43.641246    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused"
May 28 00:20:43.641499 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:43.641313    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused"
May 28 00:20:47.920616 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:47.920581    2576 generic.go:358] "Generic (PLEG): container finished" podID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerID="8efb37107b35aea1f1ff146457b13842ce52855d1771550df9fd2276c619f8aa" exitCode=0
May 28 00:20:47.920980 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:47.920649    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" event={"ID":"7bc38c66-9b71-4d63-a1be-b83d30a52103","Type":"ContainerDied","Data":"8efb37107b35aea1f1ff146457b13842ce52855d1771550df9fd2276c619f8aa"}
May 28 00:20:48.929101 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:48.929054    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" event={"ID":"7bc38c66-9b71-4d63-a1be-b83d30a52103","Type":"ContainerStarted","Data":"aea1f27d7995cc74b9fc4beae20ca4f85bf16f11f073ad636239375f89ec19be"}
May 28 00:20:48.929590 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:48.929107    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" event={"ID":"7bc38c66-9b71-4d63-a1be-b83d30a52103","Type":"ContainerStarted","Data":"d64c833c38e04a74344d1b6e05340d28fe3634f82445dd58bca68d3f22c269e6"}
May 28 00:20:48.929590 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:48.929124    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" event={"ID":"7bc38c66-9b71-4d63-a1be-b83d30a52103","Type":"ContainerStarted","Data":"2cce09254454466b6268728d43895e33e8e7f31ebe41a1876f16fac287281dfc"}
May 28 00:20:48.929590 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:48.929139    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" event={"ID":"7bc38c66-9b71-4d63-a1be-b83d30a52103","Type":"ContainerStarted","Data":"92164ba7030837498edf924b3ed5c64b6c010b879054147dc15d97f39acc44c8"}
May 28 00:20:48.955799 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:48.955253    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" podStartSLOduration=13.955231526 podStartE2EDuration="13.955231526s" podCreationTimestamp="2026-05-28 00:20:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:20:48.953346891 +0000 UTC m=+568.574378905" watchObservedRunningTime="2026-05-28 00:20:48.955231526 +0000 UTC m=+568.576263541"
May 28 00:20:51.425352 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:51.424299    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:51.425352 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:51.424359    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:51.425352 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:51.424373    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:51.425352 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:51.424386    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:51.425995 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:51.425908    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.30:8816: connect: connection refused"
May 28 00:20:51.426163 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:51.426122    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused"
May 28 00:20:51.426300 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:51.426151    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused"
May 28 00:20:51.426300 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:51.426131    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.30:6571: connect: connection refused"
May 28 00:20:53.641417 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:53.641368    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused"
May 28 00:20:53.641883 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:53.641381    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused"
May 28 00:20:54.425797 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:54.425763    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:54.425979 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:54.425928    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:54.426291 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:54.426262    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:54.426429 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:54.426299    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:54.426429 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:54.426313    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:54.426429 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:54.426325    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:54.426663 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:54.426641    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:54.426719 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:54.426704    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:54.426898 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:54.426884    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:54.430326 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:54.430306    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:54.954783 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:54.954751    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:20:54.957937 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:20:54.957913    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:21:03.641783 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:03.641737    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.29:8443: connect: connection refused"
May 28 00:21:03.642263 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:03.641750    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="online" probeResult="failure" output="Get \"https://10.132.0.29:6567/health\": dial tcp 10.132.0.29:6567: connect: connection refused"
May 28 00:21:03.642263 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:03.641890    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:21:03.642263 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:03.641957    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:21:05.881815 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:05.881788    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:21:05.995768 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:05.995678    2576 generic.go:358] "Generic (PLEG): container finished" podID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerID="79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184" exitCode=137
May 28 00:21:05.995768 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:05.995704    2576 generic.go:358] "Generic (PLEG): container finished" podID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerID="50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774" exitCode=137
May 28 00:21:05.995768 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:05.995748    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" event={"ID":"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2","Type":"ContainerDied","Data":"79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184"}
May 28 00:21:05.996025 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:05.995772    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"
May 28 00:21:05.996025 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:05.995786    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" event={"ID":"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2","Type":"ContainerDied","Data":"50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774"}
May 28 00:21:05.996025 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:05.995798    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g" event={"ID":"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2","Type":"ContainerDied","Data":"4e9155e683eb0fa9a65a574b746a69df402eb8cf597e8aba5e78f93dfda14b8d"}
May 28 00:21:05.996025 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:05.995816    2576 scope.go:117] "RemoveContainer" containerID="d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba"
May 28 00:21:06.004484 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.004467    2576 scope.go:117] "RemoveContainer" containerID="79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184"
May 28 00:21:06.012077 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.012060    2576 scope.go:117] "RemoveContainer" containerID="f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf"
May 28 00:21:06.019576 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.019558    2576 scope.go:117] "RemoveContainer" containerID="50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774"
May 28 00:21:06.026981 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.026965    2576 scope.go:117] "RemoveContainer" containerID="b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb"
May 28 00:21:06.032948 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.032922    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whxgt\" (UniqueName: \"kubernetes.io/projected/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-kube-api-access-whxgt\") pod \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") "
May 28 00:21:06.033051 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.032960    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-offline-tls\") pod \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") "
May 28 00:21:06.033051 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.033009    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-online-tls\") pod \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") "
May 28 00:21:06.033051 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.033038    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-registry-tls\") pod \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") "
May 28 00:21:06.033212 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.033061    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-feast-data\") pod \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") "
May 28 00:21:06.033212 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.033113    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-ui-tls\") pod \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\" (UID: \"662869fd-dbb9-48e2-8b54-d2bdeddfe6b2\") "
May 28 00:21:06.033679 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.033650    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-feast-data" (OuterVolumeSpecName: "feast-data") pod "662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" (UID: "662869fd-dbb9-48e2-8b54-d2bdeddfe6b2"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:21:06.034719 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.034696    2576 scope.go:117] "RemoveContainer" containerID="94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28"
May 28 00:21:06.035837 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.035768    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" (UID: "662869fd-dbb9-48e2-8b54-d2bdeddfe6b2"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:21:06.035837 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.035795    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" (UID: "662869fd-dbb9-48e2-8b54-d2bdeddfe6b2"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:21:06.035947 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.035833    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-kube-api-access-whxgt" (OuterVolumeSpecName: "kube-api-access-whxgt") pod "662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" (UID: "662869fd-dbb9-48e2-8b54-d2bdeddfe6b2"). InnerVolumeSpecName "kube-api-access-whxgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:21:06.035947 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.035912    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-online-tls" (OuterVolumeSpecName: "online-tls") pod "662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" (UID: "662869fd-dbb9-48e2-8b54-d2bdeddfe6b2"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:21:06.035947 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.035936    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" (UID: "662869fd-dbb9-48e2-8b54-d2bdeddfe6b2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:21:06.048483 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.048462    2576 scope.go:117] "RemoveContainer" containerID="d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba"
May 28 00:21:06.048809 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:21:06.048785    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba\": container with ID starting with d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba not found: ID does not exist" containerID="d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba"
May 28 00:21:06.048874 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.048817    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba"} err="failed to get container status \"d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba\": rpc error: code = NotFound desc = could not find container \"d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba\": container with ID starting with d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba not found: ID does not exist"
May 28 00:21:06.048874 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.048838    2576 scope.go:117] "RemoveContainer" containerID="79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184"
May 28 00:21:06.049090 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:21:06.049072    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184\": container with ID starting with 79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184 not found: ID does not exist" containerID="79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184"
May 28 00:21:06.049146 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.049101    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184"} err="failed to get container status \"79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184\": rpc error: code = NotFound desc = could not find container \"79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184\": container with ID starting with 79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184 not found: ID does not exist"
May 28 00:21:06.049146 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.049125    2576 scope.go:117] "RemoveContainer" containerID="f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf"
May 28 00:21:06.049372 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:21:06.049352    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf\": container with ID starting with f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf not found: ID does not exist" containerID="f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf"
May 28 00:21:06.049417 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.049379    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf"} err="failed to get container status \"f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf\": rpc error: code = NotFound desc = could not find container \"f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf\": container with ID starting with f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf not found: ID does not exist"
May 28 00:21:06.049417 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.049395    2576 scope.go:117] "RemoveContainer" containerID="50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774"
May 28 00:21:06.049619 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:21:06.049599    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774\": container with ID starting with 50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774 not found: ID does not exist" containerID="50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774"
May 28 00:21:06.049724 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.049619    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774"} err="failed to get container status \"50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774\": rpc error: code = NotFound desc = could not find container \"50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774\": container with ID starting with 50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774 not found: ID does not exist"
May 28 00:21:06.049724 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.049631    2576 scope.go:117] "RemoveContainer" containerID="b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb"
May 28 00:21:06.049864 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:21:06.049849    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb\": container with ID starting with b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb not found: ID does not exist" containerID="b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb"
May 28 00:21:06.049903 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.049868    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb"} err="failed to get container status \"b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb\": rpc error: code = NotFound desc = could not find container \"b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb\": container with ID starting with b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb not found: ID does not exist"
May 28 00:21:06.049903 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.049882    2576 scope.go:117] "RemoveContainer" containerID="94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28"
May 28 00:21:06.050116 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:21:06.050098    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28\": container with ID starting with 94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28 not found: ID does not exist" containerID="94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28"
May 28 00:21:06.050192 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.050123    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28"} err="failed to get container status \"94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28\": rpc error: code = NotFound desc = could not find container \"94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28\": container with ID starting with 94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28 not found: ID does not exist"
May 28 00:21:06.050192 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.050143    2576 scope.go:117] "RemoveContainer" containerID="d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba"
May 28 00:21:06.050361 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.050344    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba"} err="failed to get container status \"d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba\": rpc error: code = NotFound desc = could not find container \"d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba\": container with ID starting with d924b216cd9f5e531a4144d554b68955963d254e4a0468b7c8a0ad7995ffc8ba not found: ID does not exist"
May 28 00:21:06.050408 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.050363    2576 scope.go:117] "RemoveContainer" containerID="79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184"
May 28 00:21:06.050586 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.050565    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184"} err="failed to get container status \"79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184\": rpc error: code = NotFound desc = could not find container \"79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184\": container with ID starting with 79020eb62e8c37d1dfa82930f84b7d08c97c07b28fe0b0b86bf252629ea3a184 not found: ID does not exist"
May 28 00:21:06.050664 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.050587    2576 scope.go:117] "RemoveContainer" containerID="f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf"
May 28 00:21:06.050802 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.050781    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf"} err="failed to get container status \"f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf\": rpc error: code = NotFound desc = could not find container \"f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf\": container with ID starting with f463edc1214d5613e3e284bd4dfbccb8c506ee22c95d6800fde27911e613eebf not found: ID does not exist"
May 28 00:21:06.050848 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.050803    2576 scope.go:117] "RemoveContainer" containerID="50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774"
May 28 00:21:06.050990 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.050972    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774"} err="failed to get container status \"50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774\": rpc error: code = NotFound desc = could not find container \"50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774\": container with ID starting with 50ca896bd712f803506290a3443466aebd5e5cc0fccdce0b073d4ad48d01b774 not found: ID does not exist"
May 28 00:21:06.051056 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.050991    2576 scope.go:117] "RemoveContainer" containerID="b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb"
May 28 00:21:06.051201 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.051179    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb"} err="failed to get container status \"b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb\": rpc error: code = NotFound desc = could not find container \"b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb\": container with ID starting with b9fca74a4933ab08c2140109c61351836df72fd67c4504a38ed84bf4b38afeeb not found: ID does not exist"
May 28 00:21:06.051201 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.051195    2576 scope.go:117] "RemoveContainer" containerID="94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28"
May 28 00:21:06.051404 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.051387    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28"} err="failed to get container status \"94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28\": rpc error: code = NotFound desc = could not find container \"94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28\": container with ID starting with 94114df64286a410824e29afd7f95d6aacb6c83837d249766c2ec503bb909d28 not found: ID does not exist"
May 28 00:21:06.134910 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.134875    2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-ui-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:21:06.134910 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.134905    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whxgt\" (UniqueName: \"kubernetes.io/projected/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-kube-api-access-whxgt\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:21:06.134910 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.134915    2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-offline-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:21:06.135195 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.134929    2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-online-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:21:06.135195 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.134942    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-registry-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:21:06.135195 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.134954    2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2-feast-data\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:21:06.319798 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.319766    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"]
May 28 00:21:06.324050 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:06.324026    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-77j4g"]
May 28 00:21:07.016010 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:07.015976    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" path="/var/lib/kubelet/pods/662869fd-dbb9-48e2-8b54-d2bdeddfe6b2/volumes"
May 28 00:21:20.928439 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:20.928407    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:21:20.928949 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:20.928763    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:21:57.091544 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.091480    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"]
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092054    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="ui"
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092070    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="ui"
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092109    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="registry"
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092118    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="registry"
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092133    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="online"
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092141    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="online"
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092154    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="feast-apply"
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092162    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="feast-apply"
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092172    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="offline"
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092180    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="offline"
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092190    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="feast-init"
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092198    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="feast-init"
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092275    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="ui"
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092291    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="offline"
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092301    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="registry"
May 28 00:21:57.093009 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.092309    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="662869fd-dbb9-48e2-8b54-d2bdeddfe6b2" containerName="online"
May 28 00:21:57.095052 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.095031    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.099022 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.098996    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 28 00:21:57.099580 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.099565    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\""
May 28 00:21:57.099666 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.099627    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 28 00:21:57.099934 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.099918    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 28 00:21:57.100011 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.099993    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 28 00:21:57.100065 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.100035    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 28 00:21:57.100402 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.100386    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 28 00:21:57.104172 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.104149    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-wqz7v\""
May 28 00:21:57.126484 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.126449    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"]
May 28 00:21:57.141790 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.141763    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d32656e6-1f14-47a8-9943-d26b03dbd77e-feast-data\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.141940 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.141816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw8c5\" (UniqueName: \"kubernetes.io/projected/d32656e6-1f14-47a8-9943-d26b03dbd77e-kube-api-access-nw8c5\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.141940 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.141903    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/d32656e6-1f14-47a8-9943-d26b03dbd77e-registry-tls\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.141940 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.141939    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-offline-tls\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.142104 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.141975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-online-tls\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.142104 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.142031    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-ui-tls\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.242827 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.242789    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-offline-tls\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.243036 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.242851    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-online-tls\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.243036 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:21:57.242955    2576 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found
May 28 00:21:57.243036 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.242978    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-ui-tls\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.243036 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:21:57.243019    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-online-tls podName:d32656e6-1f14-47a8-9943-d26b03dbd77e nodeName:}" failed. No retries permitted until 2026-05-28 00:21:57.742997736 +0000 UTC m=+637.364029741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-online-tls") pod "feast-simple-feast-remote-setup-77f6cffd79-chjrm" (UID: "d32656e6-1f14-47a8-9943-d26b03dbd77e") : secret "feast-simple-feast-remote-setup-online-tls" not found
May 28 00:21:57.243036 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:21:57.242957    2576 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found
May 28 00:21:57.243314 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.243052    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d32656e6-1f14-47a8-9943-d26b03dbd77e-feast-data\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.243314 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:21:57.243071    2576 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found
May 28 00:21:57.243314 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:21:57.243107    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-offline-tls podName:d32656e6-1f14-47a8-9943-d26b03dbd77e nodeName:}" failed. No retries permitted until 2026-05-28 00:21:57.743087372 +0000 UTC m=+637.364119367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-offline-tls") pod "feast-simple-feast-remote-setup-77f6cffd79-chjrm" (UID: "d32656e6-1f14-47a8-9943-d26b03dbd77e") : secret "feast-simple-feast-remote-setup-offline-tls" not found
May 28 00:21:57.243314 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:21:57.243130    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-ui-tls podName:d32656e6-1f14-47a8-9943-d26b03dbd77e nodeName:}" failed. No retries permitted until 2026-05-28 00:21:57.743119658 +0000 UTC m=+637.364151653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-ui-tls") pod "feast-simple-feast-remote-setup-77f6cffd79-chjrm" (UID: "d32656e6-1f14-47a8-9943-d26b03dbd77e") : secret "feast-simple-feast-remote-setup-ui-tls" not found
May 28 00:21:57.243314 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.243163    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nw8c5\" (UniqueName: \"kubernetes.io/projected/d32656e6-1f14-47a8-9943-d26b03dbd77e-kube-api-access-nw8c5\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.243314 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.243249    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/d32656e6-1f14-47a8-9943-d26b03dbd77e-registry-tls\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.243627 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.243547    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d32656e6-1f14-47a8-9943-d26b03dbd77e-feast-data\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.243990 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.243969    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/d32656e6-1f14-47a8-9943-d26b03dbd77e-registry-tls\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.255215 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.255182    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw8c5\" (UniqueName: \"kubernetes.io/projected/d32656e6-1f14-47a8-9943-d26b03dbd77e-kube-api-access-nw8c5\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.748827 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.748791    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-offline-tls\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.749008 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.748842    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-online-tls\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.749008 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.748869    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-ui-tls\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.751429 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.751404    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-offline-tls\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.751564 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.751433    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-online-tls\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:57.751564 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:57.751489    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-ui-tls\") pod \"feast-simple-feast-remote-setup-77f6cffd79-chjrm\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:58.006612 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:58.006528    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:21:58.139945 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:58.139917    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"]
May 28 00:21:58.141864 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:21:58.141833    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd32656e6_1f14_47a8_9943_d26b03dbd77e.slice/crio-cd85968dc557bbb7402cbe6e1336d7ba19eeb3ac3f5d9d7f0e1cce6066c5b607 WatchSource:0}: Error finding container cd85968dc557bbb7402cbe6e1336d7ba19eeb3ac3f5d9d7f0e1cce6066c5b607: Status 404 returned error can't find the container with id cd85968dc557bbb7402cbe6e1336d7ba19eeb3ac3f5d9d7f0e1cce6066c5b607
May 28 00:21:58.173288 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:58.173255    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" event={"ID":"d32656e6-1f14-47a8-9943-d26b03dbd77e","Type":"ContainerStarted","Data":"cd85968dc557bbb7402cbe6e1336d7ba19eeb3ac3f5d9d7f0e1cce6066c5b607"}
May 28 00:21:59.178591 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:21:59.178547    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" event={"ID":"d32656e6-1f14-47a8-9943-d26b03dbd77e","Type":"ContainerStarted","Data":"2514f7f16fa18299c29b2ffe7db74596e3147ba66a8dabb304ed3b43dcacbb65"}
May 28 00:22:03.196386 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:03.196348    2576 generic.go:358] "Generic (PLEG): container finished" podID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerID="2514f7f16fa18299c29b2ffe7db74596e3147ba66a8dabb304ed3b43dcacbb65" exitCode=0
May 28 00:22:03.196816 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:03.196420    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" event={"ID":"d32656e6-1f14-47a8-9943-d26b03dbd77e","Type":"ContainerDied","Data":"2514f7f16fa18299c29b2ffe7db74596e3147ba66a8dabb304ed3b43dcacbb65"}
May 28 00:22:04.201564 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:04.201528    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" event={"ID":"d32656e6-1f14-47a8-9943-d26b03dbd77e","Type":"ContainerStarted","Data":"3df3c3c4861bc7e49e1d1ff002b795fb6ed1f3cd62d0997780c64c16b594ffdc"}
May 28 00:22:09.220547 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:09.220496    2576 generic.go:358] "Generic (PLEG): container finished" podID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerID="3df3c3c4861bc7e49e1d1ff002b795fb6ed1f3cd62d0997780c64c16b594ffdc" exitCode=0
May 28 00:22:09.221018 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:09.220575    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" event={"ID":"d32656e6-1f14-47a8-9943-d26b03dbd77e","Type":"ContainerDied","Data":"3df3c3c4861bc7e49e1d1ff002b795fb6ed1f3cd62d0997780c64c16b594ffdc"}
May 28 00:22:10.228210 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:10.228174    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" event={"ID":"d32656e6-1f14-47a8-9943-d26b03dbd77e","Type":"ContainerStarted","Data":"b9b3f07c57d042b2ce14b5de29039016a3629881849e0f363fa4a6152084c5c2"}
May 28 00:22:10.228210 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:10.228214    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" event={"ID":"d32656e6-1f14-47a8-9943-d26b03dbd77e","Type":"ContainerStarted","Data":"1fd20d63d97545121510a93b57d1a5b221ed2d5e3c073c62b98d555777222fd5"}
May 28 00:22:10.228747 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:10.228226    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" event={"ID":"d32656e6-1f14-47a8-9943-d26b03dbd77e","Type":"ContainerStarted","Data":"8f0d2149a215d1a9c51a4f6f28f1df68a0c85b0d8b1f14e81f617cb7dd9345be"}
May 28 00:22:10.257220 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:10.257138    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" podStartSLOduration=13.257119543 podStartE2EDuration="13.257119543s" podCreationTimestamp="2026-05-28 00:21:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:22:10.254905887 +0000 UTC m=+649.875937900" watchObservedRunningTime="2026-05-28 00:22:10.257119543 +0000 UTC m=+649.878151557"
May 28 00:22:13.007300 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:13.007262    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:22:13.007807 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:13.007405    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:22:13.007807 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:13.007423    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:22:13.009327 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:13.009288    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.31:8816: connect: connection refused"
May 28 00:22:13.009456 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:13.009316    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.31:8443: connect: connection refused"
May 28 00:22:13.009456 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:13.009317    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused"
May 28 00:22:16.007769 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:16.007737    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:22:16.008300 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:16.007860    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:22:16.008300 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:16.008265    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:22:16.008300 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:16.008299    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:22:16.008589 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:16.008569    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:22:16.008695 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:16.008610    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:22:16.011899 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:16.011877    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:22:16.252899 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:16.252867    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:22:16.255935 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:22:16.255909    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:23:18.398176 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:18.398091    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"]
May 28 00:23:18.398724 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:18.398527    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="online" containerID="cri-o://8f0d2149a215d1a9c51a4f6f28f1df68a0c85b0d8b1f14e81f617cb7dd9345be" gracePeriod=30
May 28 00:23:18.398724 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:18.398625    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="ui" containerID="cri-o://b9b3f07c57d042b2ce14b5de29039016a3629881849e0f363fa4a6152084c5c2" gracePeriod=30
May 28 00:23:18.398724 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:18.398682    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="offline" containerID="cri-o://1fd20d63d97545121510a93b57d1a5b221ed2d5e3c073c62b98d555777222fd5" gracePeriod=30
May 28 00:23:18.475925 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:18.475889    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"]
May 28 00:23:18.476381 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:18.476345    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="registry" containerID="cri-o://92164ba7030837498edf924b3ed5c64b6c010b879054147dc15d97f39acc44c8" gracePeriod=30
May 28 00:23:18.476538 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:18.476445    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="online" containerID="cri-o://2cce09254454466b6268728d43895e33e8e7f31ebe41a1876f16fac287281dfc" gracePeriod=30
May 28 00:23:18.476538 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:18.476473    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="offline" containerID="cri-o://d64c833c38e04a74344d1b6e05340d28fe3634f82445dd58bca68d3f22c269e6" gracePeriod=30
May 28 00:23:18.476741 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:18.476453    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="ui" containerID="cri-o://aea1f27d7995cc74b9fc4beae20ca4f85bf16f11f073ad636239375f89ec19be" gracePeriod=30
May 28 00:23:19.473485 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:19.473397    2576 generic.go:358] "Generic (PLEG): container finished" podID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerID="b9b3f07c57d042b2ce14b5de29039016a3629881849e0f363fa4a6152084c5c2" exitCode=0
May 28 00:23:19.473485 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:19.473422    2576 generic.go:358] "Generic (PLEG): container finished" podID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerID="8f0d2149a215d1a9c51a4f6f28f1df68a0c85b0d8b1f14e81f617cb7dd9345be" exitCode=0
May 28 00:23:19.473485 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:19.473472    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" event={"ID":"d32656e6-1f14-47a8-9943-d26b03dbd77e","Type":"ContainerDied","Data":"b9b3f07c57d042b2ce14b5de29039016a3629881849e0f363fa4a6152084c5c2"}
May 28 00:23:19.474030 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:19.473531    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" event={"ID":"d32656e6-1f14-47a8-9943-d26b03dbd77e","Type":"ContainerDied","Data":"8f0d2149a215d1a9c51a4f6f28f1df68a0c85b0d8b1f14e81f617cb7dd9345be"}
May 28 00:23:19.475890 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:19.475869    2576 generic.go:358] "Generic (PLEG): container finished" podID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerID="aea1f27d7995cc74b9fc4beae20ca4f85bf16f11f073ad636239375f89ec19be" exitCode=0
May 28 00:23:19.475890 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:19.475886    2576 generic.go:358] "Generic (PLEG): container finished" podID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerID="2cce09254454466b6268728d43895e33e8e7f31ebe41a1876f16fac287281dfc" exitCode=0
May 28 00:23:19.476023 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:19.475926    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" event={"ID":"7bc38c66-9b71-4d63-a1be-b83d30a52103","Type":"ContainerDied","Data":"aea1f27d7995cc74b9fc4beae20ca4f85bf16f11f073ad636239375f89ec19be"}
May 28 00:23:19.476023 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:19.475956    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" event={"ID":"7bc38c66-9b71-4d63-a1be-b83d30a52103","Type":"ContainerDied","Data":"2cce09254454466b6268728d43895e33e8e7f31ebe41a1876f16fac287281dfc"}
May 28 00:23:24.427403 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:24.427353    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused"
May 28 00:23:24.955825 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:24.955781    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused"
May 28 00:23:26.008435 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:26.008394    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.31:8443: connect: connection refused"
May 28 00:23:26.253708 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:26.253668    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused"
May 28 00:23:34.427083 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:34.427039    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused"
May 28 00:23:34.955601 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:34.955554    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused"
May 28 00:23:36.008877 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:36.008825    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.31:8443: connect: connection refused"
May 28 00:23:36.253708 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:36.253662    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused"
May 28 00:23:44.427463 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:44.427417    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.30:8443: connect: connection refused"
May 28 00:23:44.427992 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:44.427600    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:23:44.955397 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:44.955348    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused"
May 28 00:23:44.955625 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:44.955551    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:23:46.008780 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:46.008736    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.31:8443: connect: connection refused"
May 28 00:23:46.009238 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:46.008898    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:23:46.253810 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:46.253767    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.31:6567/health\": dial tcp 10.132.0.31:6567: connect: connection refused"
May 28 00:23:46.253993 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:46.253898    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:23:48.575708 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.575674    2576 generic.go:358] "Generic (PLEG): container finished" podID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerID="d64c833c38e04a74344d1b6e05340d28fe3634f82445dd58bca68d3f22c269e6" exitCode=137
May 28 00:23:48.575708 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.575699    2576 generic.go:358] "Generic (PLEG): container finished" podID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerID="92164ba7030837498edf924b3ed5c64b6c010b879054147dc15d97f39acc44c8" exitCode=137
May 28 00:23:48.576127 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.575742    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" event={"ID":"7bc38c66-9b71-4d63-a1be-b83d30a52103","Type":"ContainerDied","Data":"d64c833c38e04a74344d1b6e05340d28fe3634f82445dd58bca68d3f22c269e6"}
May 28 00:23:48.576127 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.575775    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" event={"ID":"7bc38c66-9b71-4d63-a1be-b83d30a52103","Type":"ContainerDied","Data":"92164ba7030837498edf924b3ed5c64b6c010b879054147dc15d97f39acc44c8"}
May 28 00:23:48.577953 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.577930    2576 generic.go:358] "Generic (PLEG): container finished" podID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerID="1fd20d63d97545121510a93b57d1a5b221ed2d5e3c073c62b98d555777222fd5" exitCode=137
May 28 00:23:48.578062 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.577969    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" event={"ID":"d32656e6-1f14-47a8-9943-d26b03dbd77e","Type":"ContainerDied","Data":"1fd20d63d97545121510a93b57d1a5b221ed2d5e3c073c62b98d555777222fd5"}
May 28 00:23:48.578062 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.577986    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm" event={"ID":"d32656e6-1f14-47a8-9943-d26b03dbd77e","Type":"ContainerDied","Data":"cd85968dc557bbb7402cbe6e1336d7ba19eeb3ac3f5d9d7f0e1cce6066c5b607"}
May 28 00:23:48.578062 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.577997    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd85968dc557bbb7402cbe6e1336d7ba19eeb3ac3f5d9d7f0e1cce6066c5b607"
May 28 00:23:48.594107 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.594086    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:23:48.747450 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.747367    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/d32656e6-1f14-47a8-9943-d26b03dbd77e-registry-tls\") pod \"d32656e6-1f14-47a8-9943-d26b03dbd77e\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") "
May 28 00:23:48.747450 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.747430    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-offline-tls\") pod \"d32656e6-1f14-47a8-9943-d26b03dbd77e\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") "
May 28 00:23:48.747683 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.747455    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-online-tls\") pod \"d32656e6-1f14-47a8-9943-d26b03dbd77e\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") "
May 28 00:23:48.747683 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.747492    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-ui-tls\") pod \"d32656e6-1f14-47a8-9943-d26b03dbd77e\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") "
May 28 00:23:48.747683 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.747545    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d32656e6-1f14-47a8-9943-d26b03dbd77e-feast-data\") pod \"d32656e6-1f14-47a8-9943-d26b03dbd77e\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") "
May 28 00:23:48.747683 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.747617    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw8c5\" (UniqueName: \"kubernetes.io/projected/d32656e6-1f14-47a8-9943-d26b03dbd77e-kube-api-access-nw8c5\") pod \"d32656e6-1f14-47a8-9943-d26b03dbd77e\" (UID: \"d32656e6-1f14-47a8-9943-d26b03dbd77e\") "
May 28 00:23:48.747889 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.747758    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32656e6-1f14-47a8-9943-d26b03dbd77e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d32656e6-1f14-47a8-9943-d26b03dbd77e" (UID: "d32656e6-1f14-47a8-9943-d26b03dbd77e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 00:23:48.747986 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.747965    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/d32656e6-1f14-47a8-9943-d26b03dbd77e-registry-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:23:48.748166 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.748134    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32656e6-1f14-47a8-9943-d26b03dbd77e-feast-data" (OuterVolumeSpecName: "feast-data") pod "d32656e6-1f14-47a8-9943-d26b03dbd77e" (UID: "d32656e6-1f14-47a8-9943-d26b03dbd77e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:23:48.749835 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.749811    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32656e6-1f14-47a8-9943-d26b03dbd77e-kube-api-access-nw8c5" (OuterVolumeSpecName: "kube-api-access-nw8c5") pod "d32656e6-1f14-47a8-9943-d26b03dbd77e" (UID: "d32656e6-1f14-47a8-9943-d26b03dbd77e"). InnerVolumeSpecName "kube-api-access-nw8c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:23:48.749922 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.749872    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "d32656e6-1f14-47a8-9943-d26b03dbd77e" (UID: "d32656e6-1f14-47a8-9943-d26b03dbd77e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:23:48.750029 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.750011    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-online-tls" (OuterVolumeSpecName: "online-tls") pod "d32656e6-1f14-47a8-9943-d26b03dbd77e" (UID: "d32656e6-1f14-47a8-9943-d26b03dbd77e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:23:48.750070 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.750033    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "d32656e6-1f14-47a8-9943-d26b03dbd77e" (UID: "d32656e6-1f14-47a8-9943-d26b03dbd77e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:23:48.849018 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.848980    2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-offline-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:23:48.849018 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.849012    2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-online-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:23:48.849018 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.849021    2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/d32656e6-1f14-47a8-9943-d26b03dbd77e-ui-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:23:48.849246 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.849030    2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/d32656e6-1f14-47a8-9943-d26b03dbd77e-feast-data\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:23:48.849246 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:48.849041    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw8c5\" (UniqueName: \"kubernetes.io/projected/d32656e6-1f14-47a8-9943-d26b03dbd77e-kube-api-access-nw8c5\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:23:49.116452 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.116429    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:23:49.252686 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.252648    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-online-tls\") pod \"7bc38c66-9b71-4d63-a1be-b83d30a52103\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") "
May 28 00:23:49.252876 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.252699    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-offline-tls\") pod \"7bc38c66-9b71-4d63-a1be-b83d30a52103\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") "
May 28 00:23:49.252876 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.252733    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc38c66-9b71-4d63-a1be-b83d30a52103-feast-data\") pod \"7bc38c66-9b71-4d63-a1be-b83d30a52103\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") "
May 28 00:23:49.252876 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.252764    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-ui-tls\") pod \"7bc38c66-9b71-4d63-a1be-b83d30a52103\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") "
May 28 00:23:49.252876 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.252797    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-registry-tls\") pod \"7bc38c66-9b71-4d63-a1be-b83d30a52103\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") "
May 28 00:23:49.252876 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.252860    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2pn6\" (UniqueName: \"kubernetes.io/projected/7bc38c66-9b71-4d63-a1be-b83d30a52103-kube-api-access-l2pn6\") pod \"7bc38c66-9b71-4d63-a1be-b83d30a52103\" (UID: \"7bc38c66-9b71-4d63-a1be-b83d30a52103\") "
May 28 00:23:49.253383 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.253346    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc38c66-9b71-4d63-a1be-b83d30a52103-feast-data" (OuterVolumeSpecName: "feast-data") pod "7bc38c66-9b71-4d63-a1be-b83d30a52103" (UID: "7bc38c66-9b71-4d63-a1be-b83d30a52103"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:23:49.255026 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.254989    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7bc38c66-9b71-4d63-a1be-b83d30a52103" (UID: "7bc38c66-9b71-4d63-a1be-b83d30a52103"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:23:49.255026 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.255011    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "7bc38c66-9b71-4d63-a1be-b83d30a52103" (UID: "7bc38c66-9b71-4d63-a1be-b83d30a52103"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:23:49.255026 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.255019    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc38c66-9b71-4d63-a1be-b83d30a52103-kube-api-access-l2pn6" (OuterVolumeSpecName: "kube-api-access-l2pn6") pod "7bc38c66-9b71-4d63-a1be-b83d30a52103" (UID: "7bc38c66-9b71-4d63-a1be-b83d30a52103"). InnerVolumeSpecName "kube-api-access-l2pn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:23:49.255290 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.255274    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "7bc38c66-9b71-4d63-a1be-b83d30a52103" (UID: "7bc38c66-9b71-4d63-a1be-b83d30a52103"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:23:49.255344 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.255293    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-online-tls" (OuterVolumeSpecName: "online-tls") pod "7bc38c66-9b71-4d63-a1be-b83d30a52103" (UID: "7bc38c66-9b71-4d63-a1be-b83d30a52103"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:23:49.354316 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.354257    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2pn6\" (UniqueName: \"kubernetes.io/projected/7bc38c66-9b71-4d63-a1be-b83d30a52103-kube-api-access-l2pn6\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:23:49.354316 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.354309    2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-online-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:23:49.354316 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.354324    2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-offline-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:23:49.354606 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.354337    2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc38c66-9b71-4d63-a1be-b83d30a52103-feast-data\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:23:49.354606 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.354349    2576 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-ui-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:23:49.354606 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.354360    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7bc38c66-9b71-4d63-a1be-b83d30a52103-registry-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:23:49.583795 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.583747    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5" event={"ID":"7bc38c66-9b71-4d63-a1be-b83d30a52103","Type":"ContainerDied","Data":"57540ae1bb2a4f4e319a0dce1726b64013d8614e987e29cbca4f036de393f72f"}
May 28 00:23:49.583795 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.583771    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"
May 28 00:23:49.583795 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.583800    2576 scope.go:117] "RemoveContainer" containerID="aea1f27d7995cc74b9fc4beae20ca4f85bf16f11f073ad636239375f89ec19be"
May 28 00:23:49.584359 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.583779    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"
May 28 00:23:49.592863 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.592845    2576 scope.go:117] "RemoveContainer" containerID="d64c833c38e04a74344d1b6e05340d28fe3634f82445dd58bca68d3f22c269e6"
May 28 00:23:49.600210 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.600189    2576 scope.go:117] "RemoveContainer" containerID="2cce09254454466b6268728d43895e33e8e7f31ebe41a1876f16fac287281dfc"
May 28 00:23:49.603972 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.603951    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"]
May 28 00:23:49.607186 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.607131    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-77f6cffd79-chjrm"]
May 28 00:23:49.609235 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.609217    2576 scope.go:117] "RemoveContainer" containerID="92164ba7030837498edf924b3ed5c64b6c010b879054147dc15d97f39acc44c8"
May 28 00:23:49.616563 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.616537    2576 scope.go:117] "RemoveContainer" containerID="8efb37107b35aea1f1ff146457b13842ce52855d1771550df9fd2276c619f8aa"
May 28 00:23:49.618239 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.618219    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"]
May 28 00:23:49.621386 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.621362    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7cf6cbc6d6-6pkd5"]
May 28 00:23:49.625138 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:49.625119    2576 scope.go:117] "RemoveContainer" containerID="b5781521310bd49652eac86956b647d91c4d224d364715446cfdd6bcfdd9ad43"
May 28 00:23:51.015950 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:51.015916    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" path="/var/lib/kubelet/pods/7bc38c66-9b71-4d63-a1be-b83d30a52103/volumes"
May 28 00:23:51.016580 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:23:51.016565    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" path="/var/lib/kubelet/pods/d32656e6-1f14-47a8-9943-d26b03dbd77e/volumes"
May 28 00:24:01.353316 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353277    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-2nbwl"]
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353685    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="feast-init"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353704    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="feast-init"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353730    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="online"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353738    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="online"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353753    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="ui"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353760    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="ui"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353770    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="feast-apply"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353778    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="feast-apply"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353786    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="online"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353794    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="online"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353805    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="registry"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353812    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="registry"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353822    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="offline"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353829    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="offline"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353841    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="feast-init"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353848    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="feast-init"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353857    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="feast-apply"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353864    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="feast-apply"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353882    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="offline"
May 28 00:24:01.353880 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353890    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="offline"
May 28 00:24:01.354852 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353904    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="ui"
May 28 00:24:01.354852 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353912    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="ui"
May 28 00:24:01.354852 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.353989    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="offline"
May 28 00:24:01.354852 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.354004    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="registry"
May 28 00:24:01.354852 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.354015    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="ui"
May 28 00:24:01.354852 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.354023    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="online"
May 28 00:24:01.354852 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.354032    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="7bc38c66-9b71-4d63-a1be-b83d30a52103" containerName="ui"
May 28 00:24:01.354852 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.354041    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="offline"
May 28 00:24:01.354852 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.354050    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="d32656e6-1f14-47a8-9943-d26b03dbd77e" containerName="online"
May 28 00:24:01.357452 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.357422    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-2nbwl"
May 28 00:24:01.359863 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.359831    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 00:24:01.359994 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.359865    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 00:24:01.359994 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.359866    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\""
May 28 00:24:01.361031 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.360998    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-tcg7q\""
May 28 00:24:01.364057 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.364031    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-2nbwl"]
May 28 00:24:01.395109 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.395076    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-854544c579-v5b6w"]
May 28 00:24:01.397245 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.397227    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-v5b6w"
May 28 00:24:01.405743 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.405716    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-v5b6w"]
May 28 00:24:01.441348 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.441311    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls6mc\" (UniqueName: \"kubernetes.io/projected/6b9a6ab8-2edb-4728-bc08-c1c208958e9d-kube-api-access-ls6mc\") pod \"postgres-7c7656cc6d-2nbwl\" (UID: \"6b9a6ab8-2edb-4728-bc08-c1c208958e9d\") " pod="test-ns-feast/postgres-7c7656cc6d-2nbwl"
May 28 00:24:01.441551 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.441354    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chvv5\" (UniqueName: \"kubernetes.io/projected/bfebca8f-84ca-442c-98a8-6771bf498c81-kube-api-access-chvv5\") pod \"redis-854544c579-v5b6w\" (UID: \"bfebca8f-84ca-442c-98a8-6771bf498c81\") " pod="test-ns-feast/redis-854544c579-v5b6w"
May 28 00:24:01.441551 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.441419    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6b9a6ab8-2edb-4728-bc08-c1c208958e9d-postgresdata\") pod \"postgres-7c7656cc6d-2nbwl\" (UID: \"6b9a6ab8-2edb-4728-bc08-c1c208958e9d\") " pod="test-ns-feast/postgres-7c7656cc6d-2nbwl"
May 28 00:24:01.541968 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.541934    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ls6mc\" (UniqueName: \"kubernetes.io/projected/6b9a6ab8-2edb-4728-bc08-c1c208958e9d-kube-api-access-ls6mc\") pod \"postgres-7c7656cc6d-2nbwl\" (UID: \"6b9a6ab8-2edb-4728-bc08-c1c208958e9d\") " pod="test-ns-feast/postgres-7c7656cc6d-2nbwl"
May 28 00:24:01.542139 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.541978    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-chvv5\" (UniqueName: \"kubernetes.io/projected/bfebca8f-84ca-442c-98a8-6771bf498c81-kube-api-access-chvv5\") pod \"redis-854544c579-v5b6w\" (UID: \"bfebca8f-84ca-442c-98a8-6771bf498c81\") " pod="test-ns-feast/redis-854544c579-v5b6w"
May 28 00:24:01.542139 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.542012    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6b9a6ab8-2edb-4728-bc08-c1c208958e9d-postgresdata\") pod \"postgres-7c7656cc6d-2nbwl\" (UID: \"6b9a6ab8-2edb-4728-bc08-c1c208958e9d\") " pod="test-ns-feast/postgres-7c7656cc6d-2nbwl"
May 28 00:24:01.542323 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.542309    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6b9a6ab8-2edb-4728-bc08-c1c208958e9d-postgresdata\") pod \"postgres-7c7656cc6d-2nbwl\" (UID: \"6b9a6ab8-2edb-4728-bc08-c1c208958e9d\") " pod="test-ns-feast/postgres-7c7656cc6d-2nbwl"
May 28 00:24:01.550970 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.550942    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chvv5\" (UniqueName: \"kubernetes.io/projected/bfebca8f-84ca-442c-98a8-6771bf498c81-kube-api-access-chvv5\") pod \"redis-854544c579-v5b6w\" (UID: \"bfebca8f-84ca-442c-98a8-6771bf498c81\") " pod="test-ns-feast/redis-854544c579-v5b6w"
May 28 00:24:01.551083 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.551007    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls6mc\" (UniqueName: \"kubernetes.io/projected/6b9a6ab8-2edb-4728-bc08-c1c208958e9d-kube-api-access-ls6mc\") pod \"postgres-7c7656cc6d-2nbwl\" (UID: \"6b9a6ab8-2edb-4728-bc08-c1c208958e9d\") " pod="test-ns-feast/postgres-7c7656cc6d-2nbwl"
May 28 00:24:01.675922 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.675820    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-2nbwl"
May 28 00:24:01.707631 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.707598    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-v5b6w"
May 28 00:24:01.814925 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.814550    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-2nbwl"]
May 28 00:24:01.817223 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:24:01.817173    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b9a6ab8_2edb_4728_bc08_c1c208958e9d.slice/crio-0ac6020cea58fe66524987dd18ac31da2c274293e1064977feb3ce830901bfb1 WatchSource:0}: Error finding container 0ac6020cea58fe66524987dd18ac31da2c274293e1064977feb3ce830901bfb1: Status 404 returned error can't find the container with id 0ac6020cea58fe66524987dd18ac31da2c274293e1064977feb3ce830901bfb1
May 28 00:24:01.819012 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.818996    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 00:24:01.841992 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:01.841969    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-v5b6w"]
May 28 00:24:01.844390 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:24:01.844361    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfebca8f_84ca_442c_98a8_6771bf498c81.slice/crio-aa5b5368374222782ca214b72901db047ccbeacd1c587964abefc1ceb2a96c51 WatchSource:0}: Error finding container aa5b5368374222782ca214b72901db047ccbeacd1c587964abefc1ceb2a96c51: Status 404 returned error can't find the container with id aa5b5368374222782ca214b72901db047ccbeacd1c587964abefc1ceb2a96c51
May 28 00:24:02.628738 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:02.628687    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-2nbwl" event={"ID":"6b9a6ab8-2edb-4728-bc08-c1c208958e9d","Type":"ContainerStarted","Data":"0ac6020cea58fe66524987dd18ac31da2c274293e1064977feb3ce830901bfb1"}
May 28 00:24:02.629840 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:02.629802    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-v5b6w" event={"ID":"bfebca8f-84ca-442c-98a8-6771bf498c81","Type":"ContainerStarted","Data":"aa5b5368374222782ca214b72901db047ccbeacd1c587964abefc1ceb2a96c51"}
May 28 00:24:03.634379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:03.634348    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-v5b6w" event={"ID":"bfebca8f-84ca-442c-98a8-6771bf498c81","Type":"ContainerStarted","Data":"b46ee24e108bb48ee3c04491acf59441d6e7b36b32c864f6452ffacb4c25c71c"}
May 28 00:24:03.652383 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:03.652331    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/redis-854544c579-v5b6w" podStartSLOduration=1.033068176 podStartE2EDuration="2.652318792s" podCreationTimestamp="2026-05-28 00:24:01 +0000 UTC" firstStartedPulling="2026-05-28 00:24:01.845939835 +0000 UTC m=+761.466971826" lastFinishedPulling="2026-05-28 00:24:03.465190452 +0000 UTC m=+763.086222442" observedRunningTime="2026-05-28 00:24:03.650732074 +0000 UTC m=+763.271764087" watchObservedRunningTime="2026-05-28 00:24:03.652318792 +0000 UTC m=+763.273350795"
May 28 00:24:08.654132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:08.654046    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-2nbwl" event={"ID":"6b9a6ab8-2edb-4728-bc08-c1c208958e9d","Type":"ContainerStarted","Data":"2ff5e9796220071018f5096a8185ee8af94ced450442f3e37135eb09a4530156"}
May 28 00:24:08.678418 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:08.678356    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/postgres-7c7656cc6d-2nbwl" podStartSLOduration=1.186932247 podStartE2EDuration="7.678338868s" podCreationTimestamp="2026-05-28 00:24:01 +0000 UTC" firstStartedPulling="2026-05-28 00:24:01.819149224 +0000 UTC m=+761.440181215" lastFinishedPulling="2026-05-28 00:24:08.310555845 +0000 UTC m=+767.931587836" observedRunningTime="2026-05-28 00:24:08.676631911 +0000 UTC m=+768.297663924" watchObservedRunningTime="2026-05-28 00:24:08.678338868 +0000 UTC m=+768.299370882"
May 28 00:24:12.074429 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.074392    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"]
May 28 00:24:12.082360 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.082335    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.087554 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.085560    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\""
May 28 00:24:12.087554 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.085776    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\""
May 28 00:24:12.087554 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.085848    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-lt26w\""
May 28 00:24:12.087554 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.086061    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\""
May 28 00:24:12.087554 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.086305    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 28 00:24:12.087554 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.087432    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"]
May 28 00:24:12.122191 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.122156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-offline-tls\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.122191 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.122190    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-feast-data\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.122448 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.122223    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-online-tls\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.122448 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.122328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99tbz\" (UniqueName: \"kubernetes.io/projected/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-kube-api-access-99tbz\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.122448 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.122373    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-registry-tls\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.223128 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.223088    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-99tbz\" (UniqueName: \"kubernetes.io/projected/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-kube-api-access-99tbz\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.223128 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.223132    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-registry-tls\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.223377 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.223187    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-offline-tls\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.223377 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.223202    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-feast-data\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.223377 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.223242    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-online-tls\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.223377 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:24:12.223329    2576 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found
May 28 00:24:12.223616 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:24:12.223403    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-online-tls podName:fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6 nodeName:}" failed. No retries permitted until 2026-05-28 00:24:12.723382379 +0000 UTC m=+772.344414370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-online-tls") pod "feast-credit-scoring-58675d75dd-57ghw" (UID: "fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6") : secret "feast-credit-scoring-online-tls" not found
May 28 00:24:12.223746 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.223724    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-feast-data\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.225762 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.225741    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-registry-tls\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.225845 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.225779    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-offline-tls\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.232278 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.232248    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99tbz\" (UniqueName: \"kubernetes.io/projected/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-kube-api-access-99tbz\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.727024 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.726983    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-online-tls\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.729393 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.729366    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-online-tls\") pod \"feast-credit-scoring-58675d75dd-57ghw\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") " pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:12.996878 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:12.996790    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:13.151132 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:13.151101    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"]
May 28 00:24:13.153217 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:24:13.153189    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7cdfa8_3be9_4ebc_bfa9_4d6df9dd2be6.slice/crio-8862dbccbd9ad4585f2a3deffd5d3bdfb9b4daf35c68ec8b8b0efe6f03c3aac2 WatchSource:0}: Error finding container 8862dbccbd9ad4585f2a3deffd5d3bdfb9b4daf35c68ec8b8b0efe6f03c3aac2: Status 404 returned error can't find the container with id 8862dbccbd9ad4585f2a3deffd5d3bdfb9b4daf35c68ec8b8b0efe6f03c3aac2
May 28 00:24:13.674779 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:13.674737    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" event={"ID":"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6","Type":"ContainerStarted","Data":"8bfe5a769bd00ef346fb46e5d9c3092478d2281dd4492b2e3a4a7b67e1e4b005"}
May 28 00:24:13.674952 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:13.674788    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" event={"ID":"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6","Type":"ContainerStarted","Data":"8862dbccbd9ad4585f2a3deffd5d3bdfb9b4daf35c68ec8b8b0efe6f03c3aac2"}
May 28 00:24:14.679037 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:14.679000    2576 generic.go:358] "Generic (PLEG): container finished" podID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerID="8bfe5a769bd00ef346fb46e5d9c3092478d2281dd4492b2e3a4a7b67e1e4b005" exitCode=0
May 28 00:24:14.679426 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:14.679079    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" event={"ID":"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6","Type":"ContainerDied","Data":"8bfe5a769bd00ef346fb46e5d9c3092478d2281dd4492b2e3a4a7b67e1e4b005"}
May 28 00:24:15.684613 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:15.684570    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" event={"ID":"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6","Type":"ContainerStarted","Data":"1e7ba86f82f26034a5ba9110dcb47e2f8a4f549297df962d476b98a228c49067"}
May 28 00:24:21.707535 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:21.707470    2576 generic.go:358] "Generic (PLEG): container finished" podID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerID="1e7ba86f82f26034a5ba9110dcb47e2f8a4f549297df962d476b98a228c49067" exitCode=0
May 28 00:24:21.707967 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:21.707540    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" event={"ID":"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6","Type":"ContainerDied","Data":"1e7ba86f82f26034a5ba9110dcb47e2f8a4f549297df962d476b98a228c49067"}
May 28 00:24:21.712207 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:21.712184    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 28 00:24:22.714737 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:22.714696    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" event={"ID":"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6","Type":"ContainerStarted","Data":"2ad1ca81ace0230271c00637b625e15bdb7dd4468d3ce01715bf1168ee4d1501"}
May 28 00:24:22.714737 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:22.714741    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" event={"ID":"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6","Type":"ContainerStarted","Data":"f5483d224655927ef84f1abca56237c85ee97dccba71f0594df7ea4242436319"}
May 28 00:24:22.715282 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:22.714750    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" event={"ID":"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6","Type":"ContainerStarted","Data":"2b4517dc4847941d1091e19261c7d80997b3d0902844872a6bc511dd789834c6"}
May 28 00:24:22.742449 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:22.742382    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" podStartSLOduration=10.742361769 podStartE2EDuration="10.742361769s" podCreationTimestamp="2026-05-28 00:24:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:24:22.738884496 +0000 UTC m=+782.359916511" watchObservedRunningTime="2026-05-28 00:24:22.742361769 +0000 UTC m=+782.363393783"
May 28 00:24:22.997298 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:22.997193    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:22.997298 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:22.997244    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:22.997298 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:22.997258    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:24.997577 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:24.997536    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:24.998056 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:24.997612    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:24.998056 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:24.997630    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:24.999592 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:24.999549    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.34:6571: connect: connection refused"
May 28 00:24:24.999731 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:24.999592    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.34:8816: connect: connection refused"
May 28 00:24:24.999731 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:24.999622    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused"
May 28 00:24:27.997656 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:27.997612    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.34:6571: connect: connection refused"
May 28 00:24:27.998641 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:27.998617    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:27.999582 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:27.999559    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:28.003362 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:28.003338    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:28.741746 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:28.741716    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:30.998185 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:30.998137    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.34:6571: connect: connection refused"
May 28 00:24:33.997753 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:33.997726    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:24:33.998464 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:24:33.998442    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:25:37.835289 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:37.835242    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"]
May 28 00:25:37.836322 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:37.836260    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="registry" containerID="cri-o://2b4517dc4847941d1091e19261c7d80997b3d0902844872a6bc511dd789834c6" gracePeriod=30
May 28 00:25:37.836951 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:37.836620    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="offline" containerID="cri-o://2ad1ca81ace0230271c00637b625e15bdb7dd4468d3ce01715bf1168ee4d1501" gracePeriod=30
May 28 00:25:37.836951 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:37.836666    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="online" containerID="cri-o://f5483d224655927ef84f1abca56237c85ee97dccba71f0594df7ea4242436319" gracePeriod=30
May 28 00:25:37.868942 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:37.868907    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-2nbwl"]
May 28 00:25:37.869174 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:37.869131    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/postgres-7c7656cc6d-2nbwl" podUID="6b9a6ab8-2edb-4728-bc08-c1c208958e9d" containerName="postgres" containerID="cri-o://2ff5e9796220071018f5096a8185ee8af94ced450442f3e37135eb09a4530156" gracePeriod=30
May 28 00:25:37.870365 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:37.870340    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-v5b6w"]
May 28 00:25:37.870540 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:37.870491    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/redis-854544c579-v5b6w" podUID="bfebca8f-84ca-442c-98a8-6771bf498c81" containerName="redis" containerID="cri-o://b46ee24e108bb48ee3c04491acf59441d6e7b36b32c864f6452ffacb4c25c71c" gracePeriod=30
May 28 00:25:37.992180 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:37.992147    2576 generic.go:358] "Generic (PLEG): container finished" podID="6b9a6ab8-2edb-4728-bc08-c1c208958e9d" containerID="2ff5e9796220071018f5096a8185ee8af94ced450442f3e37135eb09a4530156" exitCode=0
May 28 00:25:37.992311 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:37.992221    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-2nbwl" event={"ID":"6b9a6ab8-2edb-4728-bc08-c1c208958e9d","Type":"ContainerDied","Data":"2ff5e9796220071018f5096a8185ee8af94ced450442f3e37135eb09a4530156"}
May 28 00:25:37.993892 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:37.993868    2576 generic.go:358] "Generic (PLEG): container finished" podID="bfebca8f-84ca-442c-98a8-6771bf498c81" containerID="b46ee24e108bb48ee3c04491acf59441d6e7b36b32c864f6452ffacb4c25c71c" exitCode=0
May 28 00:25:37.993997 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:37.993943    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-v5b6w" event={"ID":"bfebca8f-84ca-442c-98a8-6771bf498c81","Type":"ContainerDied","Data":"b46ee24e108bb48ee3c04491acf59441d6e7b36b32c864f6452ffacb4c25c71c"}
May 28 00:25:38.142913 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:38.142891    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-2nbwl"
May 28 00:25:38.147102 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:38.147083    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-v5b6w"
May 28 00:25:38.296038 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:38.295999    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6b9a6ab8-2edb-4728-bc08-c1c208958e9d-postgresdata\") pod \"6b9a6ab8-2edb-4728-bc08-c1c208958e9d\" (UID: \"6b9a6ab8-2edb-4728-bc08-c1c208958e9d\") "
May 28 00:25:38.296238 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:38.296047    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls6mc\" (UniqueName: \"kubernetes.io/projected/6b9a6ab8-2edb-4728-bc08-c1c208958e9d-kube-api-access-ls6mc\") pod \"6b9a6ab8-2edb-4728-bc08-c1c208958e9d\" (UID: \"6b9a6ab8-2edb-4728-bc08-c1c208958e9d\") "
May 28 00:25:38.296238 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:38.296110    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chvv5\" (UniqueName: \"kubernetes.io/projected/bfebca8f-84ca-442c-98a8-6771bf498c81-kube-api-access-chvv5\") pod \"bfebca8f-84ca-442c-98a8-6771bf498c81\" (UID: \"bfebca8f-84ca-442c-98a8-6771bf498c81\") "
May 28 00:25:38.298233 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:38.298203    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9a6ab8-2edb-4728-bc08-c1c208958e9d-kube-api-access-ls6mc" (OuterVolumeSpecName: "kube-api-access-ls6mc") pod "6b9a6ab8-2edb-4728-bc08-c1c208958e9d" (UID: "6b9a6ab8-2edb-4728-bc08-c1c208958e9d"). InnerVolumeSpecName "kube-api-access-ls6mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:25:38.298349 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:38.298251    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfebca8f-84ca-442c-98a8-6771bf498c81-kube-api-access-chvv5" (OuterVolumeSpecName: "kube-api-access-chvv5") pod "bfebca8f-84ca-442c-98a8-6771bf498c81" (UID: "bfebca8f-84ca-442c-98a8-6771bf498c81"). InnerVolumeSpecName "kube-api-access-chvv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:25:38.326026 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:38.325989    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9a6ab8-2edb-4728-bc08-c1c208958e9d-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "6b9a6ab8-2edb-4728-bc08-c1c208958e9d" (UID: "6b9a6ab8-2edb-4728-bc08-c1c208958e9d"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:25:38.397652 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:38.397565    2576 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6b9a6ab8-2edb-4728-bc08-c1c208958e9d-postgresdata\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:25:38.397652 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:38.397598    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls6mc\" (UniqueName: \"kubernetes.io/projected/6b9a6ab8-2edb-4728-bc08-c1c208958e9d-kube-api-access-ls6mc\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:25:38.397652 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:38.397610    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chvv5\" (UniqueName: \"kubernetes.io/projected/bfebca8f-84ca-442c-98a8-6771bf498c81-kube-api-access-chvv5\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:25:38.739700 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:38.739597    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused"
May 28 00:25:38.999521 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:38.999426    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-2nbwl"
May 28 00:25:38.999521 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:38.999438    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-2nbwl" event={"ID":"6b9a6ab8-2edb-4728-bc08-c1c208958e9d","Type":"ContainerDied","Data":"0ac6020cea58fe66524987dd18ac31da2c274293e1064977feb3ce830901bfb1"}
May 28 00:25:38.999521 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:38.999491    2576 scope.go:117] "RemoveContainer" containerID="2ff5e9796220071018f5096a8185ee8af94ced450442f3e37135eb09a4530156"
May 28 00:25:39.002141 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:39.002117    2576 generic.go:358] "Generic (PLEG): container finished" podID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerID="f5483d224655927ef84f1abca56237c85ee97dccba71f0594df7ea4242436319" exitCode=0
May 28 00:25:39.002253 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:39.002189    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" event={"ID":"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6","Type":"ContainerDied","Data":"f5483d224655927ef84f1abca56237c85ee97dccba71f0594df7ea4242436319"}
May 28 00:25:39.003399 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:39.003375    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-v5b6w" event={"ID":"bfebca8f-84ca-442c-98a8-6771bf498c81","Type":"ContainerDied","Data":"aa5b5368374222782ca214b72901db047ccbeacd1c587964abefc1ceb2a96c51"}
May 28 00:25:39.003497 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:39.003406    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-v5b6w"
May 28 00:25:39.008420 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:39.008394    2576 scope.go:117] "RemoveContainer" containerID="b46ee24e108bb48ee3c04491acf59441d6e7b36b32c864f6452ffacb4c25c71c"
May 28 00:25:39.021709 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:39.021685    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-2nbwl"]
May 28 00:25:39.025240 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:39.025219    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-2nbwl"]
May 28 00:25:39.034802 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:39.034779    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-v5b6w"]
May 28 00:25:39.040003 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:39.039983    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-854544c579-v5b6w"]
May 28 00:25:41.015756 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:41.015724    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b9a6ab8-2edb-4728-bc08-c1c208958e9d" path="/var/lib/kubelet/pods/6b9a6ab8-2edb-4728-bc08-c1c208958e9d/volumes"
May 28 00:25:41.016193 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:41.016044    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfebca8f-84ca-442c-98a8-6771bf498c81" path="/var/lib/kubelet/pods/bfebca8f-84ca-442c-98a8-6771bf498c81/volumes"
May 28 00:25:48.738977 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:48.738938    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused"
May 28 00:25:58.739002 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:58.738957    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused"
May 28 00:25:58.739395 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:25:58.739094    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:26:07.999625 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:07.999582    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.34:8816: connect: connection refused"
May 28 00:26:08.113975 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.113884    2576 generic.go:358] "Generic (PLEG): container finished" podID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerID="2ad1ca81ace0230271c00637b625e15bdb7dd4468d3ce01715bf1168ee4d1501" exitCode=137
May 28 00:26:08.113975 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.113916    2576 generic.go:358] "Generic (PLEG): container finished" podID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerID="2b4517dc4847941d1091e19261c7d80997b3d0902844872a6bc511dd789834c6" exitCode=137
May 28 00:26:08.113975 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.113937    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" event={"ID":"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6","Type":"ContainerDied","Data":"2ad1ca81ace0230271c00637b625e15bdb7dd4468d3ce01715bf1168ee4d1501"}
May 28 00:26:08.113975 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.113959    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" event={"ID":"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6","Type":"ContainerDied","Data":"2b4517dc4847941d1091e19261c7d80997b3d0902844872a6bc511dd789834c6"}
May 28 00:26:08.474897 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.474873    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:26:08.565098 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.565061    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-online-tls\") pod \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") "
May 28 00:26:08.565098 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.565104    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99tbz\" (UniqueName: \"kubernetes.io/projected/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-kube-api-access-99tbz\") pod \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") "
May 28 00:26:08.565357 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.565183    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-offline-tls\") pod \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") "
May 28 00:26:08.565357 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.565205    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-feast-data\") pod \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") "
May 28 00:26:08.565357 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.565228    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-registry-tls\") pod \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\" (UID: \"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6\") "
May 28 00:26:08.567315 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.567278    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-online-tls" (OuterVolumeSpecName: "online-tls") pod "fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" (UID: "fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:26:08.567442 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.567362    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-kube-api-access-99tbz" (OuterVolumeSpecName: "kube-api-access-99tbz") pod "fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" (UID: "fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6"). InnerVolumeSpecName "kube-api-access-99tbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:26:08.567442 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.567362    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" (UID: "fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:26:08.567619 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.567601    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" (UID: "fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:26:08.574131 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.574102    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-feast-data" (OuterVolumeSpecName: "feast-data") pod "fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" (UID: "fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:26:08.666320 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.666214    2576 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-offline-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:26:08.666320 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.666261    2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-feast-data\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:26:08.666320 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.666270    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-registry-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:26:08.666320 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.666279    2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-online-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:26:08.666320 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:08.666289    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99tbz\" (UniqueName: \"kubernetes.io/projected/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6-kube-api-access-99tbz\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:26:09.119598 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:09.119560    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw" event={"ID":"fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6","Type":"ContainerDied","Data":"8862dbccbd9ad4585f2a3deffd5d3bdfb9b4daf35c68ec8b8b0efe6f03c3aac2"}
May 28 00:26:09.120031 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:09.119611    2576 scope.go:117] "RemoveContainer" containerID="2ad1ca81ace0230271c00637b625e15bdb7dd4468d3ce01715bf1168ee4d1501"
May 28 00:26:09.120031 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:09.119573    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"
May 28 00:26:09.128356 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:09.128332    2576 scope.go:117] "RemoveContainer" containerID="f5483d224655927ef84f1abca56237c85ee97dccba71f0594df7ea4242436319"
May 28 00:26:09.136085 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:09.136054    2576 scope.go:117] "RemoveContainer" containerID="2b4517dc4847941d1091e19261c7d80997b3d0902844872a6bc511dd789834c6"
May 28 00:26:09.140458 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:09.140434    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"]
May 28 00:26:09.143171 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:09.143149    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-58675d75dd-57ghw"]
May 28 00:26:09.145563 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:09.145546    2576 scope.go:117] "RemoveContainer" containerID="1e7ba86f82f26034a5ba9110dcb47e2f8a4f549297df962d476b98a228c49067"
May 28 00:26:09.153286 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:09.153265    2576 scope.go:117] "RemoveContainer" containerID="8bfe5a769bd00ef346fb46e5d9c3092478d2281dd4492b2e3a4a7b67e1e4b005"
May 28 00:26:11.016236 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:11.016164    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" path="/var/lib/kubelet/pods/fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6/volumes"
May 28 00:26:20.954200 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:20.954175    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:26:20.955068 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:26:20.955047    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:27:59.017864 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.017830    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-854544c579-q9cwv"]
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018200    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfebca8f-84ca-442c-98a8-6771bf498c81" containerName="redis"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018213    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfebca8f-84ca-442c-98a8-6771bf498c81" containerName="redis"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018222    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b9a6ab8-2edb-4728-bc08-c1c208958e9d" containerName="postgres"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018227    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b9a6ab8-2edb-4728-bc08-c1c208958e9d" containerName="postgres"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018235    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="registry"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018240    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="registry"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018249    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="online"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018254    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="online"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018262    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="feast-apply"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018267    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="feast-apply"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018277    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="feast-init"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018281    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="feast-init"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018290    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="offline"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018296    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="offline"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018351    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="bfebca8f-84ca-442c-98a8-6771bf498c81" containerName="redis"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018358    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="registry"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018366    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="6b9a6ab8-2edb-4728-bc08-c1c208958e9d" containerName="postgres"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018373    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="online"
May 28 00:27:59.018438 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.018381    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="fa7cdfa8-3be9-4ebc-bfa9-4d6df9dd2be6" containerName="offline"
May 28 00:27:59.021316 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.021295    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-q9cwv"
May 28 00:27:59.024376 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.024355    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\""
May 28 00:27:59.024491 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.024387    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\""
May 28 00:27:59.025578 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.025562    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-c4ltx\""
May 28 00:27:59.033937 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.033913    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-q9cwv"]
May 28 00:27:59.103432 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.103398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kq26\" (UniqueName: \"kubernetes.io/projected/02134b9a-546b-4c2c-b665-27a604335af3-kube-api-access-8kq26\") pod \"redis-854544c579-q9cwv\" (UID: \"02134b9a-546b-4c2c-b665-27a604335af3\") " pod="test-ns-feast-rest/redis-854544c579-q9cwv"
May 28 00:27:59.204625 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.204571    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8kq26\" (UniqueName: \"kubernetes.io/projected/02134b9a-546b-4c2c-b665-27a604335af3-kube-api-access-8kq26\") pod \"redis-854544c579-q9cwv\" (UID: \"02134b9a-546b-4c2c-b665-27a604335af3\") " pod="test-ns-feast-rest/redis-854544c579-q9cwv"
May 28 00:27:59.213735 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.213710    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kq26\" (UniqueName: \"kubernetes.io/projected/02134b9a-546b-4c2c-b665-27a604335af3-kube-api-access-8kq26\") pod \"redis-854544c579-q9cwv\" (UID: \"02134b9a-546b-4c2c-b665-27a604335af3\") " pod="test-ns-feast-rest/redis-854544c579-q9cwv"
May 28 00:27:59.342355 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.342316    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-q9cwv"
May 28 00:27:59.466795 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:27:59.466734    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-q9cwv"]
May 28 00:27:59.469074 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:27:59.469042    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02134b9a_546b_4c2c_b665_27a604335af3.slice/crio-3e455320ef5efd74617d852d5a893ef80ec89a98c65c70a483be516921b8febe WatchSource:0}: Error finding container 3e455320ef5efd74617d852d5a893ef80ec89a98c65c70a483be516921b8febe: Status 404 returned error can't find the container with id 3e455320ef5efd74617d852d5a893ef80ec89a98c65c70a483be516921b8febe
May 28 00:28:00.472560 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:00.472523    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-q9cwv" event={"ID":"02134b9a-546b-4c2c-b665-27a604335af3","Type":"ContainerStarted","Data":"d6df3a2edea2bd0b6b994d0bb1ba279b9f615e35da8f83556407623eb66e937d"}
May 28 00:28:00.472560 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:00.472563    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-q9cwv" event={"ID":"02134b9a-546b-4c2c-b665-27a604335af3","Type":"ContainerStarted","Data":"3e455320ef5efd74617d852d5a893ef80ec89a98c65c70a483be516921b8febe"}
May 28 00:28:00.488719 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:00.488672    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/redis-854544c579-q9cwv" podStartSLOduration=1.488659582 podStartE2EDuration="1.488659582s" podCreationTimestamp="2026-05-28 00:27:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:28:00.48837866 +0000 UTC m=+1000.109410669" watchObservedRunningTime="2026-05-28 00:28:00.488659582 +0000 UTC m=+1000.109691595"
May 28 00:28:04.461203 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:04.461161    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"]
May 28 00:28:04.464696 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:04.464668    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"
May 28 00:28:04.467297 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:04.467276    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\""
May 28 00:28:04.473320 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:04.473299    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"]
May 28 00:28:04.550487 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:04.550446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/7cd47f12-49f6-43b6-a278-6f0052c872d5-postgresdata\") pod \"postgres-7c7656cc6d-6zh6f\" (UID: \"7cd47f12-49f6-43b6-a278-6f0052c872d5\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"
May 28 00:28:04.550692 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:04.550530    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64fzb\" (UniqueName: \"kubernetes.io/projected/7cd47f12-49f6-43b6-a278-6f0052c872d5-kube-api-access-64fzb\") pod \"postgres-7c7656cc6d-6zh6f\" (UID: \"7cd47f12-49f6-43b6-a278-6f0052c872d5\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"
May 28 00:28:04.651950 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:04.651910    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-64fzb\" (UniqueName: \"kubernetes.io/projected/7cd47f12-49f6-43b6-a278-6f0052c872d5-kube-api-access-64fzb\") pod \"postgres-7c7656cc6d-6zh6f\" (UID: \"7cd47f12-49f6-43b6-a278-6f0052c872d5\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"
May 28 00:28:04.652126 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:04.652005    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/7cd47f12-49f6-43b6-a278-6f0052c872d5-postgresdata\") pod \"postgres-7c7656cc6d-6zh6f\" (UID: \"7cd47f12-49f6-43b6-a278-6f0052c872d5\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"
May 28 00:28:04.652382 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:04.652364    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/7cd47f12-49f6-43b6-a278-6f0052c872d5-postgresdata\") pod \"postgres-7c7656cc6d-6zh6f\" (UID: \"7cd47f12-49f6-43b6-a278-6f0052c872d5\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"
May 28 00:28:04.662400 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:04.662376    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64fzb\" (UniqueName: \"kubernetes.io/projected/7cd47f12-49f6-43b6-a278-6f0052c872d5-kube-api-access-64fzb\") pod \"postgres-7c7656cc6d-6zh6f\" (UID: \"7cd47f12-49f6-43b6-a278-6f0052c872d5\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"
May 28 00:28:04.775971 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:04.775890    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"
May 28 00:28:04.901827 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:04.901804    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"]
May 28 00:28:04.904256 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:28:04.904226    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cd47f12_49f6_43b6_a278_6f0052c872d5.slice/crio-6ba17567643e89865a6fe05fb0a7bab9498c58039559a53d61d62955273f96c3 WatchSource:0}: Error finding container 6ba17567643e89865a6fe05fb0a7bab9498c58039559a53d61d62955273f96c3: Status 404 returned error can't find the container with id 6ba17567643e89865a6fe05fb0a7bab9498c58039559a53d61d62955273f96c3
May 28 00:28:05.489301 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:05.489259    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f" event={"ID":"7cd47f12-49f6-43b6-a278-6f0052c872d5","Type":"ContainerStarted","Data":"dde2d3d5a311cb76b20863cb1906035f92def107a4368653a84ba94f96280656"}
May 28 00:28:05.489716 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:05.489308    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f" event={"ID":"7cd47f12-49f6-43b6-a278-6f0052c872d5","Type":"ContainerStarted","Data":"6ba17567643e89865a6fe05fb0a7bab9498c58039559a53d61d62955273f96c3"}
May 28 00:28:05.506537 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:05.506464    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f" podStartSLOduration=1.5064515630000002 podStartE2EDuration="1.506451563s" podCreationTimestamp="2026-05-28 00:28:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:28:05.505684783 +0000 UTC m=+1005.126716797" watchObservedRunningTime="2026-05-28 00:28:05.506451563 +0000 UTC m=+1005.127483575"
May 28 00:28:10.105350 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.105317    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"]
May 28 00:28:10.109013 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.108992    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:10.111890 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.111761    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\""
May 28 00:28:10.111890 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.111762    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\""
May 28 00:28:10.111890 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.111770    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-c9z8w\""
May 28 00:28:10.111890 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.111805    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\""
May 28 00:28:10.119446 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.119423    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"]
May 28 00:28:10.202873 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.202835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/805173e5-053c-464e-81b7-da1596de1a1e-online-tls\") pod \"feast-credit-scoring-6c98954979-xw55s\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") " pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:10.202873 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.202883    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/805173e5-053c-464e-81b7-da1596de1a1e-registry-tls\") pod \"feast-credit-scoring-6c98954979-xw55s\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") " pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:10.203129 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.203033    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfc69\" (UniqueName: \"kubernetes.io/projected/805173e5-053c-464e-81b7-da1596de1a1e-kube-api-access-vfc69\") pod \"feast-credit-scoring-6c98954979-xw55s\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") " pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:10.203129 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.203080    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/805173e5-053c-464e-81b7-da1596de1a1e-feast-data\") pod \"feast-credit-scoring-6c98954979-xw55s\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") " pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:10.303578 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.303545    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/805173e5-053c-464e-81b7-da1596de1a1e-registry-tls\") pod \"feast-credit-scoring-6c98954979-xw55s\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") " pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:10.303758 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.303640    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vfc69\" (UniqueName: \"kubernetes.io/projected/805173e5-053c-464e-81b7-da1596de1a1e-kube-api-access-vfc69\") pod \"feast-credit-scoring-6c98954979-xw55s\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") " pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:10.303758 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.303666    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/805173e5-053c-464e-81b7-da1596de1a1e-feast-data\") pod \"feast-credit-scoring-6c98954979-xw55s\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") " pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:10.303758 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.303693    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/805173e5-053c-464e-81b7-da1596de1a1e-online-tls\") pod \"feast-credit-scoring-6c98954979-xw55s\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") " pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:10.303864 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:28:10.303785    2576 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found
May 28 00:28:10.303864 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:28:10.303846    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/805173e5-053c-464e-81b7-da1596de1a1e-online-tls podName:805173e5-053c-464e-81b7-da1596de1a1e nodeName:}" failed. No retries permitted until 2026-05-28 00:28:10.803826539 +0000 UTC m=+1010.424858543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/805173e5-053c-464e-81b7-da1596de1a1e-online-tls") pod "feast-credit-scoring-6c98954979-xw55s" (UID: "805173e5-053c-464e-81b7-da1596de1a1e") : secret "feast-credit-scoring-online-tls" not found
May 28 00:28:10.304136 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.304105    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/805173e5-053c-464e-81b7-da1596de1a1e-feast-data\") pod \"feast-credit-scoring-6c98954979-xw55s\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") " pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:10.306026 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.306001    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/805173e5-053c-464e-81b7-da1596de1a1e-registry-tls\") pod \"feast-credit-scoring-6c98954979-xw55s\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") " pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:10.313124 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.313100    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfc69\" (UniqueName: \"kubernetes.io/projected/805173e5-053c-464e-81b7-da1596de1a1e-kube-api-access-vfc69\") pod \"feast-credit-scoring-6c98954979-xw55s\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") " pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:10.807626 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.807588    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/805173e5-053c-464e-81b7-da1596de1a1e-online-tls\") pod \"feast-credit-scoring-6c98954979-xw55s\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") " pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:10.810014 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:10.809987    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/805173e5-053c-464e-81b7-da1596de1a1e-online-tls\") pod \"feast-credit-scoring-6c98954979-xw55s\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") " pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:11.019322 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:11.019291    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:11.148478 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:11.148438    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"]
May 28 00:28:11.151290 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:28:11.151259    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805173e5_053c_464e_81b7_da1596de1a1e.slice/crio-8d0bae9c0232574f589b87dc101f8510f63084d5a20aeda61b392b3cdd689f89 WatchSource:0}: Error finding container 8d0bae9c0232574f589b87dc101f8510f63084d5a20aeda61b392b3cdd689f89: Status 404 returned error can't find the container with id 8d0bae9c0232574f589b87dc101f8510f63084d5a20aeda61b392b3cdd689f89
May 28 00:28:11.509444 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:11.509356    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" event={"ID":"805173e5-053c-464e-81b7-da1596de1a1e","Type":"ContainerStarted","Data":"3d67dd9c348f6bbc1882ccc194304599f9f0f561f29dd3c8b5998c0770f346ce"}
May 28 00:28:11.509444 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:11.509393    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" event={"ID":"805173e5-053c-464e-81b7-da1596de1a1e","Type":"ContainerStarted","Data":"8d0bae9c0232574f589b87dc101f8510f63084d5a20aeda61b392b3cdd689f89"}
May 28 00:28:13.520379 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:13.520350    2576 generic.go:358] "Generic (PLEG): container finished" podID="805173e5-053c-464e-81b7-da1596de1a1e" containerID="3d67dd9c348f6bbc1882ccc194304599f9f0f561f29dd3c8b5998c0770f346ce" exitCode=0
May 28 00:28:13.520790 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:13.520440    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" event={"ID":"805173e5-053c-464e-81b7-da1596de1a1e","Type":"ContainerDied","Data":"3d67dd9c348f6bbc1882ccc194304599f9f0f561f29dd3c8b5998c0770f346ce"}
May 28 00:28:14.526313 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:14.526271    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" event={"ID":"805173e5-053c-464e-81b7-da1596de1a1e","Type":"ContainerStarted","Data":"4d8e1b4418bc20882ac9ff569b5b0fe5cfa8e1b25e818bccb60a9b84089d5b9f"}
May 28 00:28:21.002010 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:21.001914    2576 scope.go:117] "RemoveContainer" containerID="b9b3f07c57d042b2ce14b5de29039016a3629881849e0f363fa4a6152084c5c2"
May 28 00:28:21.010196 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:21.010173    2576 scope.go:117] "RemoveContainer" containerID="1fd20d63d97545121510a93b57d1a5b221ed2d5e3c073c62b98d555777222fd5"
May 28 00:28:21.018092 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:21.018073    2576 scope.go:117] "RemoveContainer" containerID="3df3c3c4861bc7e49e1d1ff002b795fb6ed1f3cd62d0997780c64c16b594ffdc"
May 28 00:28:21.025748 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:21.025732    2576 scope.go:117] "RemoveContainer" containerID="2514f7f16fa18299c29b2ffe7db74596e3147ba66a8dabb304ed3b43dcacbb65"
May 28 00:28:21.032451 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:21.032426    2576 scope.go:117] "RemoveContainer" containerID="8f0d2149a215d1a9c51a4f6f28f1df68a0c85b0d8b1f14e81f617cb7dd9345be"
May 28 00:28:21.554306 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:21.554271    2576 generic.go:358] "Generic (PLEG): container finished" podID="805173e5-053c-464e-81b7-da1596de1a1e" containerID="4d8e1b4418bc20882ac9ff569b5b0fe5cfa8e1b25e818bccb60a9b84089d5b9f" exitCode=0
May 28 00:28:21.554540 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:21.554311    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" event={"ID":"805173e5-053c-464e-81b7-da1596de1a1e","Type":"ContainerDied","Data":"4d8e1b4418bc20882ac9ff569b5b0fe5cfa8e1b25e818bccb60a9b84089d5b9f"}
May 28 00:28:21.559321 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:21.559305    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\""
May 28 00:28:22.560841 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:22.560798    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" event={"ID":"805173e5-053c-464e-81b7-da1596de1a1e","Type":"ContainerStarted","Data":"736ced4b435d009ea52232de05556b28bfd13c35ca46e7c865e497cd0b035362"}
May 28 00:28:22.561221 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:22.560853    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" event={"ID":"805173e5-053c-464e-81b7-da1596de1a1e","Type":"ContainerStarted","Data":"955c6e47756c74f2c58bdf28a7347cc3b3b0e4af7e8c93eb3b4e3914cd8a1581"}
May 28 00:28:22.584144 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:22.584074    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" podStartSLOduration=12.584053271 podStartE2EDuration="12.584053271s" podCreationTimestamp="2026-05-28 00:28:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:28:22.582172448 +0000 UTC m=+1022.203204485" watchObservedRunningTime="2026-05-28 00:28:22.584053271 +0000 UTC m=+1022.205085285"
May 28 00:28:23.019816 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:23.019733    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:23.019816 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:23.019777    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:23.021480 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:23.021440    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused"
May 28 00:28:23.021480 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:23.021466    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused"
May 28 00:28:26.019908 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:26.019859    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused"
May 28 00:28:26.019908 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:26.019885    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused"
May 28 00:28:29.020087 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:29.020043    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused"
May 28 00:28:29.025460 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:29.025431    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:29.026052 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:29.026031    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:29.028787 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:29.028764    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:31.019693 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:31.019650    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:32.020744 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:32.020696    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.37:6571: connect: connection refused"
May 28 00:28:35.020969 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:35.020942    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:35.021673 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:35.021650    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:28:35.927282 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:35.927250    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"]
May 28 00:28:35.932740 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:35.932715    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:35.935304 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:35.935274    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\""
May 28 00:28:35.935304 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:35.935292    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\""
May 28 00:28:35.935472 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:35.935311    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-65cv6\""
May 28 00:28:35.936181 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:35.936164    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\""
May 28 00:28:35.941820 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:35.941800    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"]
May 28 00:28:36.025948 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.025914    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-online-tls\") pod \"feast-driver-ranking-6664b6f44-bdrcm\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") " pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:36.026314 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.025969    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-feast-data\") pod \"feast-driver-ranking-6664b6f44-bdrcm\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") " pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:36.026314 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.026029    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97n4r\" (UniqueName: \"kubernetes.io/projected/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-kube-api-access-97n4r\") pod \"feast-driver-ranking-6664b6f44-bdrcm\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") " pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:36.026314 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.026069    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-registry-tls\") pod \"feast-driver-ranking-6664b6f44-bdrcm\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") " pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:36.126810 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.126775    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-online-tls\") pod \"feast-driver-ranking-6664b6f44-bdrcm\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") " pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:36.127003 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.126977    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-feast-data\") pod \"feast-driver-ranking-6664b6f44-bdrcm\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") " pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:36.127073 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.127060    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-97n4r\" (UniqueName: \"kubernetes.io/projected/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-kube-api-access-97n4r\") pod \"feast-driver-ranking-6664b6f44-bdrcm\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") " pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:36.127128 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.127097    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-registry-tls\") pod \"feast-driver-ranking-6664b6f44-bdrcm\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") " pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:36.127580 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.127557    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-feast-data\") pod \"feast-driver-ranking-6664b6f44-bdrcm\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") " pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:36.129426 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.129408    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-online-tls\") pod \"feast-driver-ranking-6664b6f44-bdrcm\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") " pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:36.129498 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.129479    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-registry-tls\") pod \"feast-driver-ranking-6664b6f44-bdrcm\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") " pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:36.135523 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.135482    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97n4r\" (UniqueName: \"kubernetes.io/projected/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-kube-api-access-97n4r\") pod \"feast-driver-ranking-6664b6f44-bdrcm\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") " pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:36.243623 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.243535    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:36.370533 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.370491    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"]
May 28 00:28:36.372444 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:28:36.372418    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504a2c2d_40d7_4fae_b664_0e1b8f81ac5c.slice/crio-1553cd02059e6f50577a77a66bbadda5062eaed8b1cc404aa0d3451ca2667385 WatchSource:0}: Error finding container 1553cd02059e6f50577a77a66bbadda5062eaed8b1cc404aa0d3451ca2667385: Status 404 returned error can't find the container with id 1553cd02059e6f50577a77a66bbadda5062eaed8b1cc404aa0d3451ca2667385
May 28 00:28:36.613811 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.612983    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" event={"ID":"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c","Type":"ContainerStarted","Data":"33d11a1823902fec2414d69a9f76e7e0b973122d744058a9b419f9973dcd521e"}
May 28 00:28:36.613811 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:36.613032    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" event={"ID":"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c","Type":"ContainerStarted","Data":"1553cd02059e6f50577a77a66bbadda5062eaed8b1cc404aa0d3451ca2667385"}
May 28 00:28:41.632626 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:41.632591    2576 generic.go:358] "Generic (PLEG): container finished" podID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerID="33d11a1823902fec2414d69a9f76e7e0b973122d744058a9b419f9973dcd521e" exitCode=0
May 28 00:28:41.633008 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:41.632664    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" event={"ID":"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c","Type":"ContainerDied","Data":"33d11a1823902fec2414d69a9f76e7e0b973122d744058a9b419f9973dcd521e"}
May 28 00:28:42.638595 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:42.638560    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" event={"ID":"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c","Type":"ContainerStarted","Data":"33e232fbf1092975a0763ed05b68377554060bad57b1cfa1e63327241fcf8830"}
May 28 00:28:49.666889 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:49.666855    2576 generic.go:358] "Generic (PLEG): container finished" podID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerID="33e232fbf1092975a0763ed05b68377554060bad57b1cfa1e63327241fcf8830" exitCode=0
May 28 00:28:49.667263 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:49.666925    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" event={"ID":"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c","Type":"ContainerDied","Data":"33e232fbf1092975a0763ed05b68377554060bad57b1cfa1e63327241fcf8830"}
May 28 00:28:50.674494 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:50.674454    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" event={"ID":"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c","Type":"ContainerStarted","Data":"5f0ebe1663f3e9aa10c33d5b2fd14247e37925ab209991f83c1b4bbbe774998a"}
May 28 00:28:50.674494 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:50.674492    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" event={"ID":"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c","Type":"ContainerStarted","Data":"2e5a148f6a4516a5a714989ada6752f1c433f42e38d705df63cbee50008bc809"}
May 28 00:28:50.701315 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:50.701259    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" podStartSLOduration=15.701239199 podStartE2EDuration="15.701239199s" podCreationTimestamp="2026-05-28 00:28:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:28:50.699486606 +0000 UTC m=+1050.320518644" watchObservedRunningTime="2026-05-28 00:28:50.701239199 +0000 UTC m=+1050.322271218"
May 28 00:28:51.244713 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:51.244673    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:51.244901 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:51.244729    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:51.246383 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:51.246332    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.38:6571: connect: connection refused"
May 28 00:28:51.246532 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:51.246384    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused"
May 28 00:28:54.244883 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:54.244829    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.38:6571: connect: connection refused"
May 28 00:28:54.245284 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:54.244884    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused"
May 28 00:28:56.243829 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:56.243784    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:56.243829 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:56.243827    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:57.244825 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:57.244764    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.38:6571: connect: connection refused"
May 28 00:28:57.250156 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:57.250131    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:28:57.254325 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:28:57.254295    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:29:00.244818 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:29:00.244781    2576 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.38:6571: connect: connection refused"
May 28 00:29:03.244555 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:29:03.244501    2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:29:03.250913 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:29:03.250883    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:30:20.529382 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:20.529347    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"]
May 28 00:30:20.529995 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:20.529752    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="registry" containerID="cri-o://955c6e47756c74f2c58bdf28a7347cc3b3b0e4af7e8c93eb3b4e3914cd8a1581" gracePeriod=30
May 28 00:30:20.529995 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:20.529862    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="online" containerID="cri-o://736ced4b435d009ea52232de05556b28bfd13c35ca46e7c865e497cd0b035362" gracePeriod=30
May 28 00:30:20.538115 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:20.538089    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"]
May 28 00:30:20.538371 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:20.538348    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="registry" containerID="cri-o://2e5a148f6a4516a5a714989ada6752f1c433f42e38d705df63cbee50008bc809" gracePeriod=30
May 28 00:30:20.538518 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:20.538452    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="online" containerID="cri-o://5f0ebe1663f3e9aa10c33d5b2fd14247e37925ab209991f83c1b4bbbe774998a" gracePeriod=30
May 28 00:30:22.002775 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:22.002733    2576 generic.go:358] "Generic (PLEG): container finished" podID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerID="5f0ebe1663f3e9aa10c33d5b2fd14247e37925ab209991f83c1b4bbbe774998a" exitCode=0
May 28 00:30:22.003218 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:22.002789    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" event={"ID":"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c","Type":"ContainerDied","Data":"5f0ebe1663f3e9aa10c33d5b2fd14247e37925ab209991f83c1b4bbbe774998a"}
May 28 00:30:22.004882 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:22.004860    2576 generic.go:358] "Generic (PLEG): container finished" podID="805173e5-053c-464e-81b7-da1596de1a1e" containerID="736ced4b435d009ea52232de05556b28bfd13c35ca46e7c865e497cd0b035362" exitCode=0
May 28 00:30:22.004998 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:22.004934    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" event={"ID":"805173e5-053c-464e-81b7-da1596de1a1e","Type":"ContainerDied","Data":"736ced4b435d009ea52232de05556b28bfd13c35ca46e7c865e497cd0b035362"}
May 28 00:30:22.780241 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:22.780200    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"]
May 28 00:30:22.780425 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:22.780392    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f" podUID="7cd47f12-49f6-43b6-a278-6f0052c872d5" containerName="postgres" containerID="cri-o://dde2d3d5a311cb76b20863cb1906035f92def107a4368653a84ba94f96280656" gracePeriod=30
May 28 00:30:22.787831 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:22.787794    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-q9cwv"]
May 28 00:30:22.788077 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:22.788049    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/redis-854544c579-q9cwv" podUID="02134b9a-546b-4c2c-b665-27a604335af3" containerName="redis" containerID="cri-o://d6df3a2edea2bd0b6b994d0bb1ba279b9f615e35da8f83556407623eb66e937d" gracePeriod=30
May 28 00:30:23.009965 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:23.009930    2576 generic.go:358] "Generic (PLEG): container finished" podID="7cd47f12-49f6-43b6-a278-6f0052c872d5" containerID="dde2d3d5a311cb76b20863cb1906035f92def107a4368653a84ba94f96280656" exitCode=0
May 28 00:30:23.010344 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:23.010002    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f" event={"ID":"7cd47f12-49f6-43b6-a278-6f0052c872d5","Type":"ContainerDied","Data":"dde2d3d5a311cb76b20863cb1906035f92def107a4368653a84ba94f96280656"}
May 28 00:30:23.011603 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:23.011575    2576 generic.go:358] "Generic (PLEG): container finished" podID="02134b9a-546b-4c2c-b665-27a604335af3" containerID="d6df3a2edea2bd0b6b994d0bb1ba279b9f615e35da8f83556407623eb66e937d" exitCode=0
May 28 00:30:23.015876 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:23.015850    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-q9cwv" event={"ID":"02134b9a-546b-4c2c-b665-27a604335af3","Type":"ContainerDied","Data":"d6df3a2edea2bd0b6b994d0bb1ba279b9f615e35da8f83556407623eb66e937d"}
May 28 00:30:23.028534 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:23.028499    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"
May 28 00:30:23.062035 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:23.062011    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-q9cwv"
May 28 00:30:23.075010 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:23.074984    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/7cd47f12-49f6-43b6-a278-6f0052c872d5-postgresdata\") pod \"7cd47f12-49f6-43b6-a278-6f0052c872d5\" (UID: \"7cd47f12-49f6-43b6-a278-6f0052c872d5\") "
May 28 00:30:23.075159 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:23.075047    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64fzb\" (UniqueName: \"kubernetes.io/projected/7cd47f12-49f6-43b6-a278-6f0052c872d5-kube-api-access-64fzb\") pod \"7cd47f12-49f6-43b6-a278-6f0052c872d5\" (UID: \"7cd47f12-49f6-43b6-a278-6f0052c872d5\") "
May 28 00:30:23.077221 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:23.077193    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd47f12-49f6-43b6-a278-6f0052c872d5-kube-api-access-64fzb" (OuterVolumeSpecName: "kube-api-access-64fzb") pod "7cd47f12-49f6-43b6-a278-6f0052c872d5" (UID: "7cd47f12-49f6-43b6-a278-6f0052c872d5"). InnerVolumeSpecName "kube-api-access-64fzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:30:23.108002 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:23.107959    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd47f12-49f6-43b6-a278-6f0052c872d5-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "7cd47f12-49f6-43b6-a278-6f0052c872d5" (UID: "7cd47f12-49f6-43b6-a278-6f0052c872d5"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:30:23.175985 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:23.175950    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kq26\" (UniqueName: \"kubernetes.io/projected/02134b9a-546b-4c2c-b665-27a604335af3-kube-api-access-8kq26\") pod \"02134b9a-546b-4c2c-b665-27a604335af3\" (UID: \"02134b9a-546b-4c2c-b665-27a604335af3\") "
May 28 00:30:23.176202 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:23.176134    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64fzb\" (UniqueName: \"kubernetes.io/projected/7cd47f12-49f6-43b6-a278-6f0052c872d5-kube-api-access-64fzb\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:30:23.176202 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:23.176145    2576 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/7cd47f12-49f6-43b6-a278-6f0052c872d5-postgresdata\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:30:23.178081 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:23.178052    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02134b9a-546b-4c2c-b665-27a604335af3-kube-api-access-8kq26" (OuterVolumeSpecName: "kube-api-access-8kq26") pod "02134b9a-546b-4c2c-b665-27a604335af3" (UID: "02134b9a-546b-4c2c-b665-27a604335af3"). InnerVolumeSpecName "kube-api-access-8kq26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:30:23.276806 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:23.276761    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kq26\" (UniqueName: \"kubernetes.io/projected/02134b9a-546b-4c2c-b665-27a604335af3-kube-api-access-8kq26\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:30:24.016307 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:24.016276    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"
May 28 00:30:24.016783 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:24.016281    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-6zh6f" event={"ID":"7cd47f12-49f6-43b6-a278-6f0052c872d5","Type":"ContainerDied","Data":"6ba17567643e89865a6fe05fb0a7bab9498c58039559a53d61d62955273f96c3"}
May 28 00:30:24.016783 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:24.016389    2576 scope.go:117] "RemoveContainer" containerID="dde2d3d5a311cb76b20863cb1906035f92def107a4368653a84ba94f96280656"
May 28 00:30:24.017628 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:24.017483    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-q9cwv" event={"ID":"02134b9a-546b-4c2c-b665-27a604335af3","Type":"ContainerDied","Data":"3e455320ef5efd74617d852d5a893ef80ec89a98c65c70a483be516921b8febe"}
May 28 00:30:24.017628 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:24.017536    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-q9cwv"
May 28 00:30:24.025808 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:24.025786    2576 scope.go:117] "RemoveContainer" containerID="d6df3a2edea2bd0b6b994d0bb1ba279b9f615e35da8f83556407623eb66e937d"
May 28 00:30:24.041245 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:24.041218    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"]
May 28 00:30:24.045912 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:24.045887    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-6zh6f"]
May 28 00:30:24.057069 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:24.057043    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-q9cwv"]
May 28 00:30:24.064957 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:24.064933    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-854544c579-q9cwv"]
May 28 00:30:25.016191 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:25.016138    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02134b9a-546b-4c2c-b665-27a604335af3" path="/var/lib/kubelet/pods/02134b9a-546b-4c2c-b665-27a604335af3/volumes"
May 28 00:30:25.016625 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:25.016603    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cd47f12-49f6-43b6-a278-6f0052c872d5" path="/var/lib/kubelet/pods/7cd47f12-49f6-43b6-a278-6f0052c872d5/volumes"
May 28 00:30:27.251321 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:27.251274    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused"
May 28 00:30:29.026893 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:29.026854    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused"
May 28 00:30:37.251841 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:37.251798    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused"
May 28 00:30:39.026683 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:39.026642    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused"
May 28 00:30:47.251818 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:47.251725    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused"
May 28 00:30:47.252198 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:47.251858    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:30:49.026666 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:49.026625    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused"
May 28 00:30:49.027085 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:49.026753    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:30:51.116002 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.115970    2576 generic.go:358] "Generic (PLEG): container finished" podID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerID="2e5a148f6a4516a5a714989ada6752f1c433f42e38d705df63cbee50008bc809" exitCode=137
May 28 00:30:51.116455 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.116039    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" event={"ID":"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c","Type":"ContainerDied","Data":"2e5a148f6a4516a5a714989ada6752f1c433f42e38d705df63cbee50008bc809"}
May 28 00:30:51.118154 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.118131    2576 generic.go:358] "Generic (PLEG): container finished" podID="805173e5-053c-464e-81b7-da1596de1a1e" containerID="955c6e47756c74f2c58bdf28a7347cc3b3b0e4af7e8c93eb3b4e3914cd8a1581" exitCode=137
May 28 00:30:51.118261 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.118165    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" event={"ID":"805173e5-053c-464e-81b7-da1596de1a1e","Type":"ContainerDied","Data":"955c6e47756c74f2c58bdf28a7347cc3b3b0e4af7e8c93eb3b4e3914cd8a1581"}
May 28 00:30:51.196113 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.196082    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:30:51.199957 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.199937    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:30:51.227723 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.227690    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/805173e5-053c-464e-81b7-da1596de1a1e-feast-data\") pod \"805173e5-053c-464e-81b7-da1596de1a1e\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") "
May 28 00:30:51.227858 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.227730    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-online-tls\") pod \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") "
May 28 00:30:51.227858 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.227785    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-registry-tls\") pod \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") "
May 28 00:30:51.227858 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.227827    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-feast-data\") pod \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") "
May 28 00:30:51.228034 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.227874    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/805173e5-053c-464e-81b7-da1596de1a1e-registry-tls\") pod \"805173e5-053c-464e-81b7-da1596de1a1e\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") "
May 28 00:30:51.228034 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.227927    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97n4r\" (UniqueName: \"kubernetes.io/projected/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-kube-api-access-97n4r\") pod \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\" (UID: \"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c\") "
May 28 00:30:51.228034 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.227960    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfc69\" (UniqueName: \"kubernetes.io/projected/805173e5-053c-464e-81b7-da1596de1a1e-kube-api-access-vfc69\") pod \"805173e5-053c-464e-81b7-da1596de1a1e\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") "
May 28 00:30:51.228034 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.227994    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/805173e5-053c-464e-81b7-da1596de1a1e-online-tls\") pod \"805173e5-053c-464e-81b7-da1596de1a1e\" (UID: \"805173e5-053c-464e-81b7-da1596de1a1e\") "
May 28 00:30:51.228347 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.228289    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-feast-data" (OuterVolumeSpecName: "feast-data") pod "504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" (UID: "504a2c2d-40d7-4fae-b664-0e1b8f81ac5c"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:30:51.230471 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.230445    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805173e5-053c-464e-81b7-da1596de1a1e-online-tls" (OuterVolumeSpecName: "online-tls") pod "805173e5-053c-464e-81b7-da1596de1a1e" (UID: "805173e5-053c-464e-81b7-da1596de1a1e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:30:51.230806 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.230774    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805173e5-053c-464e-81b7-da1596de1a1e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "805173e5-053c-464e-81b7-da1596de1a1e" (UID: "805173e5-053c-464e-81b7-da1596de1a1e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:30:51.230877 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.230810    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" (UID: "504a2c2d-40d7-4fae-b664-0e1b8f81ac5c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:30:51.230877 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.230837    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-kube-api-access-97n4r" (OuterVolumeSpecName: "kube-api-access-97n4r") pod "504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" (UID: "504a2c2d-40d7-4fae-b664-0e1b8f81ac5c"). InnerVolumeSpecName "kube-api-access-97n4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:30:51.231038 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.231015    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-online-tls" (OuterVolumeSpecName: "online-tls") pod "504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" (UID: "504a2c2d-40d7-4fae-b664-0e1b8f81ac5c"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 00:30:51.231175 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.231117    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805173e5-053c-464e-81b7-da1596de1a1e-kube-api-access-vfc69" (OuterVolumeSpecName: "kube-api-access-vfc69") pod "805173e5-053c-464e-81b7-da1596de1a1e" (UID: "805173e5-053c-464e-81b7-da1596de1a1e"). InnerVolumeSpecName "kube-api-access-vfc69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:30:51.238846 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.238820    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805173e5-053c-464e-81b7-da1596de1a1e-feast-data" (OuterVolumeSpecName: "feast-data") pod "805173e5-053c-464e-81b7-da1596de1a1e" (UID: "805173e5-053c-464e-81b7-da1596de1a1e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:30:51.329062 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.328961    2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/805173e5-053c-464e-81b7-da1596de1a1e-feast-data\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:30:51.329062 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.329004    2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-online-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:30:51.329062 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.329014    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-registry-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:30:51.329062 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.329022    2576 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-feast-data\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:30:51.329062 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.329030    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/805173e5-053c-464e-81b7-da1596de1a1e-registry-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:30:51.329062 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.329040    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97n4r\" (UniqueName: \"kubernetes.io/projected/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c-kube-api-access-97n4r\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:30:51.329062 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.329051    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfc69\" (UniqueName: \"kubernetes.io/projected/805173e5-053c-464e-81b7-da1596de1a1e-kube-api-access-vfc69\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:30:51.329062 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:51.329060    2576 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/805173e5-053c-464e-81b7-da1596de1a1e-online-tls\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:30:52.123896 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.123860    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm" event={"ID":"504a2c2d-40d7-4fae-b664-0e1b8f81ac5c","Type":"ContainerDied","Data":"1553cd02059e6f50577a77a66bbadda5062eaed8b1cc404aa0d3451ca2667385"}
May 28 00:30:52.123896 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.123872    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"
May 28 00:30:52.124620 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.123908    2576 scope.go:117] "RemoveContainer" containerID="5f0ebe1663f3e9aa10c33d5b2fd14247e37925ab209991f83c1b4bbbe774998a"
May 28 00:30:52.126066 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.126043    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s" event={"ID":"805173e5-053c-464e-81b7-da1596de1a1e","Type":"ContainerDied","Data":"8d0bae9c0232574f589b87dc101f8510f63084d5a20aeda61b392b3cdd689f89"}
May 28 00:30:52.126179 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.126086    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"
May 28 00:30:52.133342 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.133303    2576 scope.go:117] "RemoveContainer" containerID="2e5a148f6a4516a5a714989ada6752f1c433f42e38d705df63cbee50008bc809"
May 28 00:30:52.147323 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.147302    2576 scope.go:117] "RemoveContainer" containerID="33e232fbf1092975a0763ed05b68377554060bad57b1cfa1e63327241fcf8830"
May 28 00:30:52.154776 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.154753    2576 scope.go:117] "RemoveContainer" containerID="33d11a1823902fec2414d69a9f76e7e0b973122d744058a9b419f9973dcd521e"
May 28 00:30:52.157518 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.157483    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"]
May 28 00:30:52.162854 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.162801    2576 scope.go:117] "RemoveContainer" containerID="736ced4b435d009ea52232de05556b28bfd13c35ca46e7c865e497cd0b035362"
May 28 00:30:52.164820 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.164796    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-6664b6f44-bdrcm"]
May 28 00:30:52.170674 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.170653    2576 scope.go:117] "RemoveContainer" containerID="955c6e47756c74f2c58bdf28a7347cc3b3b0e4af7e8c93eb3b4e3914cd8a1581"
May 28 00:30:52.178392 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.178338    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"]
May 28 00:30:52.178451 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.178427    2576 scope.go:117] "RemoveContainer" containerID="4d8e1b4418bc20882ac9ff569b5b0fe5cfa8e1b25e818bccb60a9b84089d5b9f"
May 28 00:30:52.185857 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.185833    2576 scope.go:117] "RemoveContainer" containerID="3d67dd9c348f6bbc1882ccc194304599f9f0f561f29dd3c8b5998c0770f346ce"
May 28 00:30:52.189226 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:52.189207    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-6c98954979-xw55s"]
May 28 00:30:53.017057 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:53.017024    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" path="/var/lib/kubelet/pods/504a2c2d-40d7-4fae-b664-0e1b8f81ac5c/volumes"
May 28 00:30:53.017554 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:30:53.017541    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="805173e5-053c-464e-81b7-da1596de1a1e" path="/var/lib/kubelet/pods/805173e5-053c-464e-81b7-da1596de1a1e/volumes"
May 28 00:31:20.978114 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:31:20.978080    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:31:20.980609 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:31:20.980587    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:36:21.002895 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:36:21.002859    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:36:21.007420 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:36:21.007390    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:41:10.755443 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.755351    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fjwx8/must-gather-tz82f"]
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.755875    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="feast-init"
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.755895    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="feast-init"
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.755910    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="feast-init"
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.755920    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="feast-init"
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.755932    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="registry"
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.755941    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="registry"
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.755957    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="registry"
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.755966    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="registry"
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.755983    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02134b9a-546b-4c2c-b665-27a604335af3" containerName="redis"
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.755992    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="02134b9a-546b-4c2c-b665-27a604335af3" containerName="redis"
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756002    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd47f12-49f6-43b6-a278-6f0052c872d5" containerName="postgres"
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756010    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd47f12-49f6-43b6-a278-6f0052c872d5" containerName="postgres"
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756029    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="feast-apply"
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756037    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="feast-apply"
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756049    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="online"
May 28 00:41:10.756059 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756059    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="online"
May 28 00:41:10.756872 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756071    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="feast-apply"
May 28 00:41:10.756872 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756080    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="feast-apply"
May 28 00:41:10.756872 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756105    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="online"
May 28 00:41:10.756872 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756115    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="online"
May 28 00:41:10.756872 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756202    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="online"
May 28 00:41:10.756872 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756219    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="02134b9a-546b-4c2c-b665-27a604335af3" containerName="redis"
May 28 00:41:10.756872 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756230    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="online"
May 28 00:41:10.756872 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756241    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="7cd47f12-49f6-43b6-a278-6f0052c872d5" containerName="postgres"
May 28 00:41:10.756872 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756252    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="805173e5-053c-464e-81b7-da1596de1a1e" containerName="registry"
May 28 00:41:10.756872 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.756265    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="504a2c2d-40d7-4fae-b664-0e1b8f81ac5c" containerName="registry"
May 28 00:41:10.759714 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.759693    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjwx8/must-gather-tz82f"
May 28 00:41:10.762383 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.762365    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fjwx8\"/\"kube-root-ca.crt\""
May 28 00:41:10.763472 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.763444    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fjwx8\"/\"openshift-service-ca.crt\""
May 28 00:41:10.763472 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.763474    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fjwx8\"/\"default-dockercfg-2mslx\""
May 28 00:41:10.765419 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.765390    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fjwx8/must-gather-tz82f"]
May 28 00:41:10.888564 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.888526    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjvzm\" (UniqueName: \"kubernetes.io/projected/fbc08a46-1d5d-4977-92fc-5dd66495fefa-kube-api-access-cjvzm\") pod \"must-gather-tz82f\" (UID: \"fbc08a46-1d5d-4977-92fc-5dd66495fefa\") " pod="openshift-must-gather-fjwx8/must-gather-tz82f"
May 28 00:41:10.888564 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.888567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fbc08a46-1d5d-4977-92fc-5dd66495fefa-must-gather-output\") pod \"must-gather-tz82f\" (UID: \"fbc08a46-1d5d-4977-92fc-5dd66495fefa\") " pod="openshift-must-gather-fjwx8/must-gather-tz82f"
May 28 00:41:10.989274 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.989236    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cjvzm\" (UniqueName: \"kubernetes.io/projected/fbc08a46-1d5d-4977-92fc-5dd66495fefa-kube-api-access-cjvzm\") pod \"must-gather-tz82f\" (UID: \"fbc08a46-1d5d-4977-92fc-5dd66495fefa\") " pod="openshift-must-gather-fjwx8/must-gather-tz82f"
May 28 00:41:10.989274 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.989275    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fbc08a46-1d5d-4977-92fc-5dd66495fefa-must-gather-output\") pod \"must-gather-tz82f\" (UID: \"fbc08a46-1d5d-4977-92fc-5dd66495fefa\") " pod="openshift-must-gather-fjwx8/must-gather-tz82f"
May 28 00:41:10.989618 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.989602    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fbc08a46-1d5d-4977-92fc-5dd66495fefa-must-gather-output\") pod \"must-gather-tz82f\" (UID: \"fbc08a46-1d5d-4977-92fc-5dd66495fefa\") " pod="openshift-must-gather-fjwx8/must-gather-tz82f"
May 28 00:41:10.997900 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:10.997878    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjvzm\" (UniqueName: \"kubernetes.io/projected/fbc08a46-1d5d-4977-92fc-5dd66495fefa-kube-api-access-cjvzm\") pod \"must-gather-tz82f\" (UID: \"fbc08a46-1d5d-4977-92fc-5dd66495fefa\") " pod="openshift-must-gather-fjwx8/must-gather-tz82f"
May 28 00:41:11.078709 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:11.078664    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjwx8/must-gather-tz82f"
May 28 00:41:11.203973 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:11.203947    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fjwx8/must-gather-tz82f"]
May 28 00:41:11.206621 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:41:11.206589    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbc08a46_1d5d_4977_92fc_5dd66495fefa.slice/crio-7c3b9ea0aaa331e8171517b195d101bc95baa202d74097f7439757a4068ce45d WatchSource:0}: Error finding container 7c3b9ea0aaa331e8171517b195d101bc95baa202d74097f7439757a4068ce45d: Status 404 returned error can't find the container with id 7c3b9ea0aaa331e8171517b195d101bc95baa202d74097f7439757a4068ce45d
May 28 00:41:11.208847 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:11.208823    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 00:41:12.039030 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:12.038963    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjwx8/must-gather-tz82f" event={"ID":"fbc08a46-1d5d-4977-92fc-5dd66495fefa","Type":"ContainerStarted","Data":"7c3b9ea0aaa331e8171517b195d101bc95baa202d74097f7439757a4068ce45d"}
May 28 00:41:18.061247 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:18.061206    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjwx8/must-gather-tz82f" event={"ID":"fbc08a46-1d5d-4977-92fc-5dd66495fefa","Type":"ContainerStarted","Data":"8bd9021637dfaf584b1223500dcb4ab781ba01f477423c7e7fdaf091d85bd46c"}
May 28 00:41:18.061714 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:18.061265    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjwx8/must-gather-tz82f" event={"ID":"fbc08a46-1d5d-4977-92fc-5dd66495fefa","Type":"ContainerStarted","Data":"28fbde9fa8aa9110d373ab9da2e25a885d7392c98ffb308170c18b2dc31d9ee8"}
May 28 00:41:18.077295 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:18.077233    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-fjwx8/must-gather-tz82f" podStartSLOduration=2.072581029 podStartE2EDuration="8.077217521s" podCreationTimestamp="2026-05-28 00:41:10 +0000 UTC" firstStartedPulling="2026-05-28 00:41:11.208953465 +0000 UTC m=+1790.829985456" lastFinishedPulling="2026-05-28 00:41:17.213589954 +0000 UTC m=+1796.834621948" observedRunningTime="2026-05-28 00:41:18.076800638 +0000 UTC m=+1797.697832653" watchObservedRunningTime="2026-05-28 00:41:18.077217521 +0000 UTC m=+1797.698249536"
May 28 00:41:21.031718 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:21.031685    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:41:21.035483 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:21.035461    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:41:27.091635 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:27.091599    2576 generic.go:358] "Generic (PLEG): container finished" podID="fbc08a46-1d5d-4977-92fc-5dd66495fefa" containerID="28fbde9fa8aa9110d373ab9da2e25a885d7392c98ffb308170c18b2dc31d9ee8" exitCode=0
May 28 00:41:27.092085 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:27.091678    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjwx8/must-gather-tz82f" event={"ID":"fbc08a46-1d5d-4977-92fc-5dd66495fefa","Type":"ContainerDied","Data":"28fbde9fa8aa9110d373ab9da2e25a885d7392c98ffb308170c18b2dc31d9ee8"}
May 28 00:41:27.092085 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:27.092019    2576 scope.go:117] "RemoveContainer" containerID="28fbde9fa8aa9110d373ab9da2e25a885d7392c98ffb308170c18b2dc31d9ee8"
May 28 00:41:27.977206 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:27.977172    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-fjwx8_must-gather-tz82f_fbc08a46-1d5d-4977-92fc-5dd66495fefa/gather/0.log"
May 28 00:41:31.270573 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:31.270540    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h7v4h_3a977914-7378-4daf-a7ca-b1e54548a760/global-pull-secret-syncer/0.log"
May 28 00:41:31.451543 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:31.451497    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jxqqz_4ec0d0ea-61c0-4df1-9eb0-a78662baca1d/konnectivity-agent/0.log"
May 28 00:41:31.473991 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:31.473966    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-163.ec2.internal_276f8146c60adf430fac9496a6d82374/haproxy/0.log"
May 28 00:41:33.292169 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:33.292131    2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-fjwx8/must-gather-tz82f"]
May 28 00:41:33.292585 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:33.292374    2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-fjwx8/must-gather-tz82f" podUID="fbc08a46-1d5d-4977-92fc-5dd66495fefa" containerName="copy" containerID="cri-o://8bd9021637dfaf584b1223500dcb4ab781ba01f477423c7e7fdaf091d85bd46c" gracePeriod=2
May 28 00:41:33.297570 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:33.297546    2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-fjwx8/must-gather-tz82f"]
May 28 00:41:33.525335 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:33.525313    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-fjwx8_must-gather-tz82f_fbc08a46-1d5d-4977-92fc-5dd66495fefa/copy/0.log"
May 28 00:41:33.525700 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:33.525683    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjwx8/must-gather-tz82f"
May 28 00:41:33.528006 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:33.527981    2576 status_manager.go:895] "Failed to get status for pod" podUID="fbc08a46-1d5d-4977-92fc-5dd66495fefa" pod="openshift-must-gather-fjwx8/must-gather-tz82f" err="pods \"must-gather-tz82f\" is forbidden: User \"system:node:ip-10-0-128-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-fjwx8\": no relationship found between node 'ip-10-0-128-163.ec2.internal' and this object"
May 28 00:41:33.592057 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:33.591970    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fbc08a46-1d5d-4977-92fc-5dd66495fefa-must-gather-output\") pod \"fbc08a46-1d5d-4977-92fc-5dd66495fefa\" (UID: \"fbc08a46-1d5d-4977-92fc-5dd66495fefa\") "
May 28 00:41:33.592057 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:33.592021    2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjvzm\" (UniqueName: \"kubernetes.io/projected/fbc08a46-1d5d-4977-92fc-5dd66495fefa-kube-api-access-cjvzm\") pod \"fbc08a46-1d5d-4977-92fc-5dd66495fefa\" (UID: \"fbc08a46-1d5d-4977-92fc-5dd66495fefa\") "
May 28 00:41:33.592336 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:33.592312    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc08a46-1d5d-4977-92fc-5dd66495fefa-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "fbc08a46-1d5d-4977-92fc-5dd66495fefa" (UID: "fbc08a46-1d5d-4977-92fc-5dd66495fefa"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 00:41:33.594210 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:33.594191    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc08a46-1d5d-4977-92fc-5dd66495fefa-kube-api-access-cjvzm" (OuterVolumeSpecName: "kube-api-access-cjvzm") pod "fbc08a46-1d5d-4977-92fc-5dd66495fefa" (UID: "fbc08a46-1d5d-4977-92fc-5dd66495fefa"). InnerVolumeSpecName "kube-api-access-cjvzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 00:41:33.692850 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:33.692813    2576 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fbc08a46-1d5d-4977-92fc-5dd66495fefa-must-gather-output\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:41:33.692850 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:33.692842    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjvzm\" (UniqueName: \"kubernetes.io/projected/fbc08a46-1d5d-4977-92fc-5dd66495fefa-kube-api-access-cjvzm\") on node \"ip-10-0-128-163.ec2.internal\" DevicePath \"\""
May 28 00:41:34.113888 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.113863    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-fjwx8_must-gather-tz82f_fbc08a46-1d5d-4977-92fc-5dd66495fefa/copy/0.log"
May 28 00:41:34.114203 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.114176    2576 generic.go:358] "Generic (PLEG): container finished" podID="fbc08a46-1d5d-4977-92fc-5dd66495fefa" containerID="8bd9021637dfaf584b1223500dcb4ab781ba01f477423c7e7fdaf091d85bd46c" exitCode=143
May 28 00:41:34.114263 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.114231    2576 scope.go:117] "RemoveContainer" containerID="8bd9021637dfaf584b1223500dcb4ab781ba01f477423c7e7fdaf091d85bd46c"
May 28 00:41:34.114306 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.114230    2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjwx8/must-gather-tz82f"
May 28 00:41:34.117211 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.117182    2576 status_manager.go:895] "Failed to get status for pod" podUID="fbc08a46-1d5d-4977-92fc-5dd66495fefa" pod="openshift-must-gather-fjwx8/must-gather-tz82f" err="pods \"must-gather-tz82f\" is forbidden: User \"system:node:ip-10-0-128-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-fjwx8\": no relationship found between node 'ip-10-0-128-163.ec2.internal' and this object"
May 28 00:41:34.122041 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.122018    2576 scope.go:117] "RemoveContainer" containerID="28fbde9fa8aa9110d373ab9da2e25a885d7392c98ffb308170c18b2dc31d9ee8"
May 28 00:41:34.124142 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.124119    2576 status_manager.go:895] "Failed to get status for pod" podUID="fbc08a46-1d5d-4977-92fc-5dd66495fefa" pod="openshift-must-gather-fjwx8/must-gather-tz82f" err="pods \"must-gather-tz82f\" is forbidden: User \"system:node:ip-10-0-128-163.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-fjwx8\": no relationship found between node 'ip-10-0-128-163.ec2.internal' and this object"
May 28 00:41:34.133635 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.133612    2576 scope.go:117] "RemoveContainer" containerID="8bd9021637dfaf584b1223500dcb4ab781ba01f477423c7e7fdaf091d85bd46c"
May 28 00:41:34.133949 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:41:34.133930    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd9021637dfaf584b1223500dcb4ab781ba01f477423c7e7fdaf091d85bd46c\": container with ID starting with 8bd9021637dfaf584b1223500dcb4ab781ba01f477423c7e7fdaf091d85bd46c not found: ID does not exist" containerID="8bd9021637dfaf584b1223500dcb4ab781ba01f477423c7e7fdaf091d85bd46c"
May 28 00:41:34.133996 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.133962    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd9021637dfaf584b1223500dcb4ab781ba01f477423c7e7fdaf091d85bd46c"} err="failed to get container status \"8bd9021637dfaf584b1223500dcb4ab781ba01f477423c7e7fdaf091d85bd46c\": rpc error: code = NotFound desc = could not find container \"8bd9021637dfaf584b1223500dcb4ab781ba01f477423c7e7fdaf091d85bd46c\": container with ID starting with 8bd9021637dfaf584b1223500dcb4ab781ba01f477423c7e7fdaf091d85bd46c not found: ID does not exist"
May 28 00:41:34.133996 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.133984    2576 scope.go:117] "RemoveContainer" containerID="28fbde9fa8aa9110d373ab9da2e25a885d7392c98ffb308170c18b2dc31d9ee8"
May 28 00:41:34.134214 ip-10-0-128-163 kubenswrapper[2576]: E0528 00:41:34.134195    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fbde9fa8aa9110d373ab9da2e25a885d7392c98ffb308170c18b2dc31d9ee8\": container with ID starting with 28fbde9fa8aa9110d373ab9da2e25a885d7392c98ffb308170c18b2dc31d9ee8 not found: ID does not exist" containerID="28fbde9fa8aa9110d373ab9da2e25a885d7392c98ffb308170c18b2dc31d9ee8"
May 28 00:41:34.134252 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.134221    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fbde9fa8aa9110d373ab9da2e25a885d7392c98ffb308170c18b2dc31d9ee8"} err="failed to get container status \"28fbde9fa8aa9110d373ab9da2e25a885d7392c98ffb308170c18b2dc31d9ee8\": rpc error: code = NotFound desc = could not find container \"28fbde9fa8aa9110d373ab9da2e25a885d7392c98ffb308170c18b2dc31d9ee8\": container with ID starting with 28fbde9fa8aa9110d373ab9da2e25a885d7392c98ffb308170c18b2dc31d9ee8 not found: ID does not exist"
May 28 00:41:34.158412 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.158388    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_817606af-c726-4199-9fbe-6ce1ff5c6776/alertmanager/0.log"
May 28 00:41:34.187206 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.187184    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_817606af-c726-4199-9fbe-6ce1ff5c6776/config-reloader/0.log"
May 28 00:41:34.224852 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.224829    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_817606af-c726-4199-9fbe-6ce1ff5c6776/kube-rbac-proxy-web/0.log"
May 28 00:41:34.263371 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.263350    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_817606af-c726-4199-9fbe-6ce1ff5c6776/kube-rbac-proxy/0.log"
May 28 00:41:34.305319 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.305288    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_817606af-c726-4199-9fbe-6ce1ff5c6776/kube-rbac-proxy-metric/0.log"
May 28 00:41:34.331676 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.331647    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_817606af-c726-4199-9fbe-6ce1ff5c6776/prom-label-proxy/0.log"
May 28 00:41:34.357071 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.357042    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_817606af-c726-4199-9fbe-6ce1ff5c6776/init-config-reloader/0.log"
May 28 00:41:34.435980 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.435899    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-mj29t_2e9c810f-5c20-42f3-93a9-583827884992/kube-state-metrics/0.log"
May 28 00:41:34.459807 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.459782    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-mj29t_2e9c810f-5c20-42f3-93a9-583827884992/kube-rbac-proxy-main/0.log"
May 28 00:41:34.483730 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.483700    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-mj29t_2e9c810f-5c20-42f3-93a9-583827884992/kube-rbac-proxy-self/0.log"
May 28 00:41:34.510391 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.510364    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-784984c5dc-64l2b_a5b3729d-0bbc-42dc-a4b3-e0d6dbe2292e/metrics-server/0.log"
May 28 00:41:34.573041 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.573012    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9lwc6_2ef743b7-db78-4328-901c-1547852b6226/node-exporter/0.log"
May 28 00:41:34.596326 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.596304    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9lwc6_2ef743b7-db78-4328-901c-1547852b6226/kube-rbac-proxy/0.log"
May 28 00:41:34.622032 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.622010    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9lwc6_2ef743b7-db78-4328-901c-1547852b6226/init-textfile/0.log"
May 28 00:41:34.827792 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.827765    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-b9z6f_233f08a0-994d-42e8-9777-d2f4f17ba908/kube-rbac-proxy-main/0.log"
May 28 00:41:34.852990 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.852962    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-b9z6f_233f08a0-994d-42e8-9777-d2f4f17ba908/kube-rbac-proxy-self/0.log"
May 28 00:41:34.876150 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:34.876123    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-b9z6f_233f08a0-994d-42e8-9777-d2f4f17ba908/openshift-state-metrics/0.log"
May 28 00:41:35.016355 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:35.016318    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbc08a46-1d5d-4977-92fc-5dd66495fefa" path="/var/lib/kubelet/pods/fbc08a46-1d5d-4977-92fc-5dd66495fefa/volumes"
May 28 00:41:35.087559 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:35.087472    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-sjmrd_03f56a8b-5149-4ba6-a9a5-eb14ce9edf74/prometheus-operator/0.log"
May 28 00:41:35.109216 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:35.109189    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-sjmrd_03f56a8b-5149-4ba6-a9a5-eb14ce9edf74/kube-rbac-proxy/0.log"
May 28 00:41:35.136384 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:35.136352    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-4d48s_bd375ef7-7c1e-4074-8e93-6ac8f1d9920f/prometheus-operator-admission-webhook/0.log"
May 28 00:41:35.166931 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:35.166904    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b5c989bd6-wz9zb_c46be0ad-baf5-4342-ad5d-407db2a83e0c/telemeter-client/0.log"
May 28 00:41:35.192366 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:35.192342    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b5c989bd6-wz9zb_c46be0ad-baf5-4342-ad5d-407db2a83e0c/reload/0.log"
May 28 00:41:35.215415 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:35.215390    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b5c989bd6-wz9zb_c46be0ad-baf5-4342-ad5d-407db2a83e0c/kube-rbac-proxy/0.log"
May 28 00:41:35.247226 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:35.247205    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64f86464d4-qvwrw_fc9cc118-8d55-4701-b696-66e6f1816343/thanos-query/0.log"
May 28 00:41:35.269801 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:35.269770    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64f86464d4-qvwrw_fc9cc118-8d55-4701-b696-66e6f1816343/kube-rbac-proxy-web/0.log"
May 28 00:41:35.293352 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:35.293327    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64f86464d4-qvwrw_fc9cc118-8d55-4701-b696-66e6f1816343/kube-rbac-proxy/0.log"
May 28 00:41:35.316354 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:35.316328    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64f86464d4-qvwrw_fc9cc118-8d55-4701-b696-66e6f1816343/prom-label-proxy/0.log"
May 28 00:41:35.340332 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:35.340261    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64f86464d4-qvwrw_fc9cc118-8d55-4701-b696-66e6f1816343/kube-rbac-proxy-rules/0.log"
May 28 00:41:35.363196 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:35.363174    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64f86464d4-qvwrw_fc9cc118-8d55-4701-b696-66e6f1816343/kube-rbac-proxy-metrics/0.log"
May 28 00:41:36.468931 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:36.468900    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-bpdzw_3c036ea7-0bb5-45d4-bacb-3b17d1156f0b/networking-console-plugin/0.log"
May 28 00:41:36.855697 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:36.855670    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/1.log"
May 28 00:41:36.861260 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:36.861237    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nwc22_86b12396-57bb-4096-a674-699511beae9e/console-operator/2.log"
May 28 00:41:37.649997 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.649967    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-24ksn_354fa8c9-65f4-49f0-9793-12913a9b261d/volume-data-source-validator/0.log"
May 28 00:41:37.808638 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.808607    2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"]
May 28 00:41:37.809014 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.808999    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc08a46-1d5d-4977-92fc-5dd66495fefa" containerName="gather"
May 28 00:41:37.809057 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.809016    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc08a46-1d5d-4977-92fc-5dd66495fefa" containerName="gather"
May 28 00:41:37.809057 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.809026    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc08a46-1d5d-4977-92fc-5dd66495fefa" containerName="copy"
May 28 00:41:37.809057 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.809035    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc08a46-1d5d-4977-92fc-5dd66495fefa" containerName="copy"
May 28 00:41:37.809148 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.809109    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="fbc08a46-1d5d-4977-92fc-5dd66495fefa" containerName="gather"
May 28 00:41:37.809148 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.809120    2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="fbc08a46-1d5d-4977-92fc-5dd66495fefa" containerName="copy"
May 28 00:41:37.812621 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.812599    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:37.815709 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.815671    2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nc6j4\"/\"default-dockercfg-hwxt8\""
May 28 00:41:37.816592 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.816574    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nc6j4\"/\"kube-root-ca.crt\""
May 28 00:41:37.816683 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.816617    2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nc6j4\"/\"openshift-service-ca.crt\""
May 28 00:41:37.819744 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.819490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e66f1ad1-8089-495f-9a98-f8c11563d612-proc\") pod \"perf-node-gather-daemonset-vdvlt\" (UID: \"e66f1ad1-8089-495f-9a98-f8c11563d612\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:37.819744 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.819559    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vssgr\" (UniqueName: \"kubernetes.io/projected/e66f1ad1-8089-495f-9a98-f8c11563d612-kube-api-access-vssgr\") pod \"perf-node-gather-daemonset-vdvlt\" (UID: \"e66f1ad1-8089-495f-9a98-f8c11563d612\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:37.819744 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.819599    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e66f1ad1-8089-495f-9a98-f8c11563d612-sys\") pod \"perf-node-gather-daemonset-vdvlt\" (UID: \"e66f1ad1-8089-495f-9a98-f8c11563d612\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:37.819744 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.819706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e66f1ad1-8089-495f-9a98-f8c11563d612-podres\") pod \"perf-node-gather-daemonset-vdvlt\" (UID: \"e66f1ad1-8089-495f-9a98-f8c11563d612\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:37.819744 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.819738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e66f1ad1-8089-495f-9a98-f8c11563d612-lib-modules\") pod \"perf-node-gather-daemonset-vdvlt\" (UID: \"e66f1ad1-8089-495f-9a98-f8c11563d612\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:37.820253 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.820232    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"]
May 28 00:41:37.920227 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.920134    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e66f1ad1-8089-495f-9a98-f8c11563d612-podres\") pod \"perf-node-gather-daemonset-vdvlt\" (UID: \"e66f1ad1-8089-495f-9a98-f8c11563d612\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:37.920227 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.920174    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e66f1ad1-8089-495f-9a98-f8c11563d612-lib-modules\") pod \"perf-node-gather-daemonset-vdvlt\" (UID: \"e66f1ad1-8089-495f-9a98-f8c11563d612\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:37.920458 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.920230    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e66f1ad1-8089-495f-9a98-f8c11563d612-proc\") pod \"perf-node-gather-daemonset-vdvlt\" (UID: \"e66f1ad1-8089-495f-9a98-f8c11563d612\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:37.920458 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.920258    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vssgr\" (UniqueName: \"kubernetes.io/projected/e66f1ad1-8089-495f-9a98-f8c11563d612-kube-api-access-vssgr\") pod \"perf-node-gather-daemonset-vdvlt\" (UID: \"e66f1ad1-8089-495f-9a98-f8c11563d612\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:37.920458 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.920324    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e66f1ad1-8089-495f-9a98-f8c11563d612-proc\") pod \"perf-node-gather-daemonset-vdvlt\" (UID: \"e66f1ad1-8089-495f-9a98-f8c11563d612\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:37.920458 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.920325    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e66f1ad1-8089-495f-9a98-f8c11563d612-podres\") pod \"perf-node-gather-daemonset-vdvlt\" (UID: \"e66f1ad1-8089-495f-9a98-f8c11563d612\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:37.920458 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.920388    2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e66f1ad1-8089-495f-9a98-f8c11563d612-sys\") pod \"perf-node-gather-daemonset-vdvlt\" (UID: \"e66f1ad1-8089-495f-9a98-f8c11563d612\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:37.920458 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.920403    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e66f1ad1-8089-495f-9a98-f8c11563d612-lib-modules\") pod \"perf-node-gather-daemonset-vdvlt\" (UID: \"e66f1ad1-8089-495f-9a98-f8c11563d612\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:37.920771 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.920493    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e66f1ad1-8089-495f-9a98-f8c11563d612-sys\") pod \"perf-node-gather-daemonset-vdvlt\" (UID: \"e66f1ad1-8089-495f-9a98-f8c11563d612\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:37.928729 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:37.928704    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vssgr\" (UniqueName: \"kubernetes.io/projected/e66f1ad1-8089-495f-9a98-f8c11563d612-kube-api-access-vssgr\") pod \"perf-node-gather-daemonset-vdvlt\" (UID: \"e66f1ad1-8089-495f-9a98-f8c11563d612\") " pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:38.124131 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:38.124095    2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:38.251081 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:38.251036    2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"]
May 28 00:41:38.254347 ip-10-0-128-163 kubenswrapper[2576]: W0528 00:41:38.254311    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode66f1ad1_8089_495f_9a98_f8c11563d612.slice/crio-e0a79018394e648aefa024b96cb10274a1e3b1503cc7224101c6e633f119bf0a WatchSource:0}: Error finding container e0a79018394e648aefa024b96cb10274a1e3b1503cc7224101c6e633f119bf0a: Status 404 returned error can't find the container with id e0a79018394e648aefa024b96cb10274a1e3b1503cc7224101c6e633f119bf0a
May 28 00:41:38.303206 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:38.303179    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4sq87_2e2c7ab9-c05a-44d8-907a-377754c23b53/dns/0.log"
May 28 00:41:38.325285 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:38.325260    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4sq87_2e2c7ab9-c05a-44d8-907a-377754c23b53/kube-rbac-proxy/0.log"
May 28 00:41:38.504256 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:38.504180    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tmc88_d74ab08c-2275-4054-9365-ba62ea412c17/dns-node-resolver/0.log"
May 28 00:41:38.924766 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:38.924734    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mvfbv_a2f133ad-ace2-417a-a450-7f917265d112/node-ca/0.log"
May 28 00:41:39.133660 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:39.133626    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt" event={"ID":"e66f1ad1-8089-495f-9a98-f8c11563d612","Type":"ContainerStarted","Data":"a1f87361a7448a41a4ce038846cf4369c5b3e52f5ede79805fb0ce2a9ed5e50f"}
May 28 00:41:39.133660 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:39.133659    2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt" event={"ID":"e66f1ad1-8089-495f-9a98-f8c11563d612","Type":"ContainerStarted","Data":"e0a79018394e648aefa024b96cb10274a1e3b1503cc7224101c6e633f119bf0a"}
May 28 00:41:39.133871 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:39.133801    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:39.150188 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:39.150142    2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt" podStartSLOduration=2.150124872 podStartE2EDuration="2.150124872s" podCreationTimestamp="2026-05-28 00:41:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 00:41:39.148825962 +0000 UTC m=+1818.769857975" watchObservedRunningTime="2026-05-28 00:41:39.150124872 +0000 UTC m=+1818.771156888"
May 28 00:41:39.948946 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:39.948917    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tgwt7_b1715227-4b2d-44dd-a153-51973245938a/serve-healthcheck-canary/0.log"
May 28 00:41:40.278694 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:40.278614    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-wfff8_fa182962-797b-4e21-a57f-7225c6bc806a/insights-operator/0.log"
May 28 00:41:40.278694 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:40.278675    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-wfff8_fa182962-797b-4e21-a57f-7225c6bc806a/insights-operator/1.log"
May 28 00:41:40.365054 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:40.365030    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-62sqq_6e28953c-9a22-4850-941c-e9aabd07aef4/kube-rbac-proxy/0.log"
May 28 00:41:40.387436 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:40.387410    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-62sqq_6e28953c-9a22-4850-941c-e9aabd07aef4/exporter/0.log"
May 28 00:41:40.411125 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:40.411100    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-62sqq_6e28953c-9a22-4850-941c-e9aabd07aef4/extractor/0.log"
May 28 00:41:44.391042 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:44.391011    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-w8pmx_e29b5003-2cae-4189-8a90-9508ac3a7570/migrator/0.log"
May 28 00:41:44.413277 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:44.413251    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-w8pmx_e29b5003-2cae-4189-8a90-9508ac3a7570/graceful-termination/0.log"
May 28 00:41:44.693638 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:44.693560    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-j9fwb_43cdd4d1-8b9f-4b27-8280-5b668415be73/kube-storage-version-migrator-operator/1.log"
May 28 00:41:44.694392 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:44.694373    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-j9fwb_43cdd4d1-8b9f-4b27-8280-5b668415be73/kube-storage-version-migrator-operator/0.log"
May 28 00:41:45.146367 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:45.146337    2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nc6j4/perf-node-gather-daemonset-vdvlt"
May 28 00:41:45.510555 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:45.510459    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-62mcg_a2435a2d-273d-4891-918d-377147d3eb63/kube-multus/0.log"
May 28 00:41:45.693644 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:45.693616    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r9lmd_d5d293d3-caa4-44cf-bae5-219a9e493213/kube-multus-additional-cni-plugins/0.log"
May 28 00:41:45.714964 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:45.714940    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r9lmd_d5d293d3-caa4-44cf-bae5-219a9e493213/egress-router-binary-copy/0.log"
May 28 00:41:45.737251 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:45.737230    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r9lmd_d5d293d3-caa4-44cf-bae5-219a9e493213/cni-plugins/0.log"
May 28 00:41:45.758027 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:45.758004    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r9lmd_d5d293d3-caa4-44cf-bae5-219a9e493213/bond-cni-plugin/0.log"
May 28 00:41:45.781480 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:45.781458    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r9lmd_d5d293d3-caa4-44cf-bae5-219a9e493213/routeoverride-cni/0.log"
May 28 00:41:45.803144 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:45.803122    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r9lmd_d5d293d3-caa4-44cf-bae5-219a9e493213/whereabouts-cni-bincopy/0.log"
May 28 00:41:45.824048 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:45.824025    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r9lmd_d5d293d3-caa4-44cf-bae5-219a9e493213/whereabouts-cni/0.log"
May 28 00:41:46.105297 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:46.105219    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6b84s_653f150e-1c4b-4dc5-a088-6ab0ad385350/network-metrics-daemon/0.log"
May 28 00:41:46.140577 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:46.140548    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6b84s_653f150e-1c4b-4dc5-a088-6ab0ad385350/kube-rbac-proxy/0.log"
May 28 00:41:47.020944 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:47.020904    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8gct7_39bb9abb-9e4b-4adb-a92c-cb7c18563333/ovn-controller/0.log"
May 28 00:41:47.058144 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:47.058113    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8gct7_39bb9abb-9e4b-4adb-a92c-cb7c18563333/ovn-acl-logging/0.log"
May 28 00:41:47.093894 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:47.093866    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8gct7_39bb9abb-9e4b-4adb-a92c-cb7c18563333/kube-rbac-proxy-node/0.log"
May 28 00:41:47.123847 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:47.123827    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8gct7_39bb9abb-9e4b-4adb-a92c-cb7c18563333/kube-rbac-proxy-ovn-metrics/0.log"
May 28 00:41:47.147228 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:47.147203    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8gct7_39bb9abb-9e4b-4adb-a92c-cb7c18563333/northd/0.log"
May 28 00:41:47.169726 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:47.169700    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8gct7_39bb9abb-9e4b-4adb-a92c-cb7c18563333/nbdb/0.log"
May 28 00:41:47.195731 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:47.195639    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8gct7_39bb9abb-9e4b-4adb-a92c-cb7c18563333/sbdb/0.log"
May 28 00:41:47.292745 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:47.292690    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8gct7_39bb9abb-9e4b-4adb-a92c-cb7c18563333/ovnkube-controller/0.log"
May 28 00:41:49.066227 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:49.066156    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-xq6dj_2e67c0ae-9c40-46d4-91a8-a5577a46bd2f/network-check-target-container/0.log"
May 28 00:41:49.849521 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:49.849482    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-bvcpc_bd2bcc29-a5ec-4d13-832d-f74b151a4c3d/iptables-alerter/0.log"
May 28 00:41:50.553388 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:50.553361    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-zwmf5_6fe0ac68-5a7f-48ae-b7ca-bc7ffc6ceb88/tuned/0.log"
May 28 00:41:52.115481 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:52.115445    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-tr6nj_0a851a7c-3e27-429d-a713-041330a90874/cluster-samples-operator/0.log"
May 28 00:41:52.131539 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:52.131496    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-tr6nj_0a851a7c-3e27-429d-a713-041330a90874/cluster-samples-operator-watch/0.log"
May 28 00:41:53.395883 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:53.395850    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-67d667c7d-9c7dr_e299ef6a-1c2f-4ecd-8043-0f91566212b9/service-ca-controller/0.log"
May 28 00:41:53.841446 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:53.841412    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-hsfhl_637d10d6-892a-46db-ac00-41e0294451d3/csi-driver/0.log"
May 28 00:41:53.864292 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:53.864270    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-hsfhl_637d10d6-892a-46db-ac00-41e0294451d3/csi-node-driver-registrar/0.log"
May 28 00:41:53.886555 ip-10-0-128-163 kubenswrapper[2576]: I0528 00:41:53.886533    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-hsfhl_637d10d6-892a-46db-ac00-41e0294451d3/csi-liveness-probe/0.log"