May 28 11:00:28.717968 ip-10-0-133-107 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 28 11:00:28.717980 ip-10-0-133-107 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 28 11:00:28.717989 ip-10-0-133-107 systemd[1]: kubelet.service: Failed with result 'resources'.
May 28 11:00:28.718292 ip-10-0-133-107 systemd[1]: Failed to start Kubernetes Kubelet.
May 28 11:00:40.082159 ip-10-0-133-107 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 28 11:00:40.082179 ip-10-0-133-107 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 802ff430b4df4c5a982688a4bc5e76cd --
May 28 11:03:03.845436 ip-10-0-133-107 systemd[1]: Starting Kubernetes Kubelet...
May 28 11:03:04.268672 ip-10-0-133-107 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 11:03:04.268672 ip-10-0-133-107 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 28 11:03:04.268672 ip-10-0-133-107 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 11:03:04.268672 ip-10-0-133-107 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 28 11:03:04.268672 ip-10-0-133-107 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 11:03:04.269616 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.269531    2552 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 28 11:03:04.278432 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278408    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 11:03:04.278432 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278426    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 11:03:04.278432 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278431    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 11:03:04.278432 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278434    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 11:03:04.278432 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278437    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 11:03:04.278432 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278440    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278444    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278447    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278451    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278454    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278456    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278459    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278462    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278465    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278468    2552 feature_gate.go:328] unrecognized feature gate: Example2
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278472    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278475    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278478    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278484    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278487    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278490    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278493    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278495    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278498    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278500    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 11:03:04.278643 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278503    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278505    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278508    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278512    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278515    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278519    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278521    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278524    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278527    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278529    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278532    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278534    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278537    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278539    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278543    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278546    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278549    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278553    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 11:03:04.279117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278556    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278559    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278563    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278567    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278569    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278573    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278576    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278579    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278581    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278584    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278587    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278589    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278592    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278595    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278597    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278599    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278602    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278604    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278607    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278609    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 11:03:04.279605 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278612    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278614    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278617    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278619    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278623    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278625    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278628    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278631    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278634    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278636    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278639    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278641    2552 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278644    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278646    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278649    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278653    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278656    2552 feature_gate.go:328] unrecognized feature gate: Example
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278659    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278661    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278664    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 11:03:04.280100 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278666    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278669    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.278672    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279039    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279044    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279046    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279049    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279052    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279055    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279058    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279061    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279064    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279066    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279069    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279073    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279076    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279078    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279081    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279083    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279086    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 11:03:04.280592 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279089    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279092    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279094    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279097    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279100    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279103    2552 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279105    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279108    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279111    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279113    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279116    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279118    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279121    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279124    2552 feature_gate.go:328] unrecognized feature gate: Example
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279142    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279145    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279148    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279150    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279153    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279156    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 11:03:04.281067 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279158    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279161    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279163    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279166    2552 feature_gate.go:328] unrecognized feature gate: Example2
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279168    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279172    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279174    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279177    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279180    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279182    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279185    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279188    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279190    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279193    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279195    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279198    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279200    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279203    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279205    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279208    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 11:03:04.281570 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279211    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279213    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279216    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279219    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279222    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279224    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279227    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279229    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279232    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279234    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279237    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279239    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279242    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279244    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279247    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279250    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279254    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279258    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279261    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279263    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 11:03:04.282051 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279266    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279269    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279271    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279274    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279276    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279279    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279281    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279285    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.279289    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280626    2552 flags.go:64] FLAG: --address="0.0.0.0"
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280635    2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280641    2552 flags.go:64] FLAG: --anonymous-auth="true"
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280646    2552 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280650    2552 flags.go:64] FLAG: --authentication-token-webhook="false"
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280658    2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280662    2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280666    2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280670    2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280673    2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280676    2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280679    2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 28 11:03:04.282576 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280682    2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280685    2552 flags.go:64] FLAG: --cgroup-root=""
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280688    2552 flags.go:64] FLAG: --cgroups-per-qos="true"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280691    2552 flags.go:64] FLAG: --client-ca-file=""
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280694    2552 flags.go:64] FLAG: --cloud-config=""
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280697    2552 flags.go:64] FLAG: --cloud-provider="external"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280700    2552 flags.go:64] FLAG: --cluster-dns="[]"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280705    2552 flags.go:64] FLAG: --cluster-domain=""
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280708    2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280711    2552 flags.go:64] FLAG: --config-dir=""
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280714    2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280717    2552 flags.go:64] FLAG: --container-log-max-files="5"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280721    2552 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280724    2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280727    2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280731    2552 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280734    2552 flags.go:64] FLAG: --contention-profiling="false"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280737    2552 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280740    2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280743    2552 flags.go:64] FLAG: --cpu-manager-policy="none"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280746    2552 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280755    2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280758    2552 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280761    2552 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280763    2552 flags.go:64] FLAG: --enable-load-reader="false"
May 28 11:03:04.283109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280769    2552 flags.go:64] FLAG: --enable-server="true"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280772    2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280777    2552 flags.go:64] FLAG: --event-burst="100"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280781    2552 flags.go:64] FLAG: --event-qps="50"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280784    2552 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280787    2552 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280790    2552 flags.go:64] FLAG: --eviction-hard=""
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280793    2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280796    2552 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280799    2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280802    2552 flags.go:64] FLAG: --eviction-soft=""
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280805    2552 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280808    2552 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280811    2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280814    2552 flags.go:64] FLAG: --experimental-mounter-path=""
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280817    2552 flags.go:64] FLAG: --fail-cgroupv1="false"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280820    2552 flags.go:64] FLAG: --fail-swap-on="true"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280823    2552 flags.go:64] FLAG: --feature-gates=""
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280827    2552 flags.go:64] FLAG: --file-check-frequency="20s"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280830    2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280832    2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280836    2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280839    2552 flags.go:64] FLAG: --healthz-port="10248"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280842    2552 flags.go:64] FLAG: --help="false"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280845    2552 flags.go:64] FLAG: --hostname-override="ip-10-0-133-107.ec2.internal"
May 28 11:03:04.283733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280848    2552 flags.go:64] FLAG: --housekeeping-interval="10s"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280850    2552 flags.go:64] FLAG: --http-check-frequency="20s"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280853    2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280857    2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280860    2552 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280863    2552 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280866    2552 flags.go:64] FLAG: --image-service-endpoint=""
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280870    2552 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280874    2552 flags.go:64] FLAG: --kube-api-burst="100"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280877    2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280880    2552 flags.go:64] FLAG: --kube-api-qps="50"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280883    2552 flags.go:64] FLAG: --kube-reserved=""
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280886    2552 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280888    2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280892    2552 flags.go:64] FLAG: --kubelet-cgroups=""
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280895    2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280897    2552 flags.go:64] FLAG: --lock-file=""
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280900    2552 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280903    2552 flags.go:64] FLAG: --log-flush-frequency="5s"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280906    2552 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280912    2552 flags.go:64] FLAG: --log-json-split-stream="false"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280915    2552 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280917    2552 flags.go:64] FLAG: --log-text-split-stream="false"
May 28 11:03:04.284352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280920    2552 flags.go:64] FLAG: --logging-format="text"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280923    2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280926    2552 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280929    2552 flags.go:64] FLAG: --manifest-url=""
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280932    2552 flags.go:64] FLAG: --manifest-url-header=""
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280936    2552 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280939    2552 flags.go:64] FLAG: --max-open-files="1000000"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280943    2552 flags.go:64] FLAG: --max-pods="110"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280946    2552 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280949    2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280952    2552 flags.go:64] FLAG: --memory-manager-policy="None"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280955    2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280958    2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280961    2552 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280964    2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280971    2552 flags.go:64] FLAG: --node-status-max-images="50"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280975    2552 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280979    2552 flags.go:64] FLAG: --oom-score-adj="-999"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280982    2552 flags.go:64] FLAG: --pod-cidr=""
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280985    2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280991    2552 flags.go:64] FLAG: --pod-manifest-path=""
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280994    2552 flags.go:64] FLAG: --pod-max-pids="-1"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.280997    2552 flags.go:64] FLAG: --pods-per-core="0"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281000    2552 flags.go:64] FLAG: --port="10250"
May 28 11:03:04.284934 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281005    2552 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281008    2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0491707ff17b9183f"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281011    2552 flags.go:64] FLAG: --qos-reserved=""
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281014    2552 flags.go:64] FLAG: --read-only-port="10255"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281017    2552 flags.go:64] FLAG: --register-node="true"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281020    2552 flags.go:64] FLAG: --register-schedulable="true"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281022    2552 flags.go:64] FLAG: --register-with-taints=""
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281026    2552 flags.go:64] FLAG: --registry-burst="10"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281029    2552 flags.go:64] FLAG: --registry-qps="5"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281032    2552 flags.go:64] FLAG: --reserved-cpus=""
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281034    2552 flags.go:64] FLAG: --reserved-memory=""
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281039    2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281042    2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281045    2552 flags.go:64] FLAG: --rotate-certificates="false"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281047    2552 flags.go:64] FLAG: --rotate-server-certificates="false"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281050    2552 flags.go:64] FLAG: --runonce="false"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281053    2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281056    2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281059    2552 flags.go:64] FLAG: --seccomp-default="false"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281062    2552 flags.go:64] FLAG: --serialize-image-pulls="true"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281065    2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281068    2552 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281071    2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281074    2552 flags.go:64] FLAG: --storage-driver-password="root"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281077    2552 flags.go:64] FLAG: --storage-driver-secure="false"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281081    2552 flags.go:64] FLAG: --storage-driver-table="stats"
May 28 11:03:04.285533 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281083    2552 flags.go:64] FLAG: --storage-driver-user="root"
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281086    2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281090    2552 flags.go:64] FLAG: --sync-frequency="1m0s"
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281092    2552 flags.go:64] FLAG: --system-cgroups=""
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281095    2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281100    2552 flags.go:64] FLAG: --system-reserved-cgroup=""
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281108    2552 flags.go:64] FLAG: --tls-cert-file=""
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281111    2552 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281115    2552 flags.go:64] FLAG: --tls-min-version=""
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281118    2552 flags.go:64] FLAG: --tls-private-key-file=""
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281120    2552 flags.go:64] FLAG: --topology-manager-policy="none"
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281123    2552 flags.go:64] FLAG: --topology-manager-policy-options=""
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281141    2552 flags.go:64] FLAG: --topology-manager-scope="container"
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281144    2552 flags.go:64] FLAG: --v="2"
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281148    2552 flags.go:64] FLAG: --version="false"
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281153    2552 flags.go:64] FLAG: --vmodule=""
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281157    2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281160    2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281251    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281255    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281258    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281262    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281266    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 11:03:04.286246 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281269    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281272    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281274    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281276    2552 feature_gate.go:328] unrecognized feature gate: Example
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281279    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281281    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281284    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281287    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281289    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281292    2552 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281294    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281298    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281301    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281304    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281306    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281310    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281313    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281316    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281318    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281320    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 11:03:04.286813 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281323    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281325    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281328    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281330    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281333    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281335    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281338    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281341    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281343    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281346    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281348    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281351    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281353    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281356    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281358    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281361    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281363    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281366    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281368    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 11:03:04.287322 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281371    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281374    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281376    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281379    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281382    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281384    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281387    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281389    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281393    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281396    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281398    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281401    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281403    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281406    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281410    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281413    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281416    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281419    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281421    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281424    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 11:03:04.287794 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281427    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281429    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281432    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281434    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281437    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281439    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281442    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281445    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281447    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281450    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281452    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281454    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281457    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281460    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281462    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281465    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281467    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281470    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281472    2552 feature_gate.go:328] unrecognized feature gate: Example2
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281475    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 11:03:04.288305 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281479    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.281481    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.281487    2552 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.287495    2552 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.287510    2552 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287557    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287562    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287566    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287570    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287573    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287577    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287579    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287582    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287585    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287588    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287590    2552 feature_gate.go:328] unrecognized feature gate: Example
May 28 11:03:04.288790 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287593    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287596    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287598    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287601    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287604    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287606    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287608    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287611    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287614    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287617    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287619    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287622    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287624    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287627    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287630    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287632    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287635    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287639    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287642    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 11:03:04.289218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287645    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287649    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287652    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287656    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287658    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287661    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287663    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287666    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287668    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287671    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287673    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287676    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287678    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287681    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287683    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287687    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287691    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287695    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287697    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 11:03:04.289705 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287700    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287703    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287705    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287708    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287711    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287713    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287716    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287719    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287721    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287724    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287726    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287729    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287731    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287734    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287738    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287741    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287744    2552 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287747    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287749    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287751    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 11:03:04.290184 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287754    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287757    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287759    2552 feature_gate.go:328] unrecognized feature gate: Example2
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287762    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287764    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287766    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287769    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287771    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287774    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287776    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287779    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287781    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287784    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287787    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287789    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287791    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 11:03:04.290679 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287794    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 11:03:04.291077 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.287799    2552 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 11:03:04.291077 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287913    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 11:03:04.291077 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287919    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 11:03:04.291077 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287922    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 11:03:04.291077 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287926    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 11:03:04.291077 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287937    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 11:03:04.291077 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287941    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 11:03:04.291077 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287944    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 11:03:04.291077 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287947    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 11:03:04.291077 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287950    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 11:03:04.291077 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287953    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 11:03:04.291077 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287956    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 11:03:04.291077 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287959    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 11:03:04.291077 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287962    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 11:03:04.291077 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287964    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287967    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287969    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287972    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287974    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287976    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287979    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287982    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287985    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287987    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287989    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287992    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287994    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.287997    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288000    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288002    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288005    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288007    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288010    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288012    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 11:03:04.291471 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288015    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288018    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288020    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288023    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288025    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288028    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288031    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288033    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288036    2552 feature_gate.go:328] unrecognized feature gate: Example2
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288038    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288041    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288043    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288046    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288048    2552 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288051    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288053    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288056    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288058    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288061    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288063    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 11:03:04.291965 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288065    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288068    2552 feature_gate.go:328] unrecognized feature gate: Example
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288070    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288073    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288075    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288078    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288080    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288082    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288085    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288087    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288089    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288092    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288095    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288098    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288100    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288104    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288107    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288110    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288113    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288116    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 11:03:04.292459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288119    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 11:03:04.292971 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288121    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 11:03:04.292971 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288124    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 11:03:04.292971 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288142    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 11:03:04.292971 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288145    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 11:03:04.292971 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288148    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 11:03:04.292971 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288150    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 11:03:04.292971 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288153    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 11:03:04.292971 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288155    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 11:03:04.292971 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288158    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 11:03:04.292971 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288160    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 11:03:04.292971 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288163    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 11:03:04.292971 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:04.288165    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 11:03:04.292971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.288170    2552 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 11:03:04.292971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.288866    2552 server.go:962] "Client rotation is on, will bootstrap in background"
May 28 11:03:04.292971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.290728    2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 28 11:03:04.293392 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.291667    2552 server.go:1019] "Starting client certificate rotation"
May 28 11:03:04.293392 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.291762    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 28 11:03:04.293392 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.291798    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 28 11:03:04.318932 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.318910    2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 28 11:03:04.321531 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.320955    2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 28 11:03:04.334332 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.334311    2552 log.go:25] "Validated CRI v1 runtime API"
May 28 11:03:04.340186 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.340169    2552 log.go:25] "Validated CRI v1 image API"
May 28 11:03:04.344194 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.344178    2552 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 28 11:03:04.346117 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.346100    2552 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 28 11:03:04.347643 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.347620    2552 fs.go:135] Filesystem UUIDs: map[73437579-196f-44d8-ad1a-7c3ff9bad287:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c9ed3ed4-d5ce-4324-b304-c2fc4cbf7e3d:/dev/nvme0n1p3]
May 28 11:03:04.347703 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.347643    2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 28 11:03:04.351921 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.351815    2552 manager.go:217] Machine: {Timestamp:2026-05-28 11:03:04.351324216 +0000 UTC m=+0.390421186 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098181 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec248e1de7da84528eea078537b86c54 SystemUUID:ec248e1d-e7da-8452-8eea-078537b86c54 BootID:802ff430-b4df-4c5a-9826-88a4bc5e76cd Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2a:ec:56:31:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2a:ec:56:31:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:bd:88:2e:0f:b1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 28 11:03:04.352003 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.351950    2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 28 11:03:04.352074 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.352062    2552 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 28 11:03:04.355026 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.354888    2552 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 28 11:03:04.355182 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.355028    2552 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-107.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 28 11:03:04.355233 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.355192    2552 topology_manager.go:138] "Creating topology manager with none policy"
May 28 11:03:04.355233 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.355201    2552 container_manager_linux.go:306] "Creating device plugin manager"
May 28 11:03:04.355233 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.355209    2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 28 11:03:04.355233 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.355223    2552 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 28 11:03:04.357009 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.356999    2552 state_mem.go:36] "Initialized new in-memory state store"
May 28 11:03:04.357112 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.357103    2552 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 28 11:03:04.359878 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.359868    2552 kubelet.go:491] "Attempting to sync node with API server"
May 28 11:03:04.359917 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.359882    2552 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 28 11:03:04.359917 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.359894    2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 28 11:03:04.359917 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.359904    2552 kubelet.go:397] "Adding apiserver pod source"
May 28 11:03:04.359917 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.359913    2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 28 11:03:04.361932 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.361913    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 28 11:03:04.362008 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.361945    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 28 11:03:04.365346 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.365329    2552 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 28 11:03:04.367795 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.367781    2552 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 28 11:03:04.368819 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.368803    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 28 11:03:04.368900 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.368830    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 28 11:03:04.368900 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.368840    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 28 11:03:04.368900 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.368855    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 28 11:03:04.368900 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.368868    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 28 11:03:04.368900 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.368877    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 28 11:03:04.368900 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.368901    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 28 11:03:04.369076 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.368910    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 28 11:03:04.369076 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.368920    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 28 11:03:04.369076 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.368929    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 28 11:03:04.369076 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.368947    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 28 11:03:04.369207 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.369155    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 28 11:03:04.369629 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.369611    2552 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 28 11:03:04.369668 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.369610    2552 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 28 11:03:04.370220 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.370210    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 28 11:03:04.370220 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.370221    2552 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 28 11:03:04.370371 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.370355    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lvl4z"
May 28 11:03:04.373156 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.373124    2552 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 28 11:03:04.373214 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.373183    2552 server.go:1295] "Started kubelet"
May 28 11:03:04.373318 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.373272    2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 28 11:03:04.373389 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.373271    2552 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 28 11:03:04.373446 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.373393    2552 server_v1.go:47] "podresources" method="list" useActivePods=true
May 28 11:03:04.373910 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.373892    2552 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-107.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 28 11:03:04.374021 ip-10-0-133-107 systemd[1]: Started Kubernetes Kubelet.
May 28 11:03:04.374929 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.374781    2552 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 28 11:03:04.376226 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.376211    2552 server.go:317] "Adding debug handlers to kubelet server"
May 28 11:03:04.380322 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.380125    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lvl4z"
May 28 11:03:04.383780 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.383760    2552 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 28 11:03:04.386582 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.386490    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 28 11:03:04.387170 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.387149    2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 28 11:03:04.387292 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.387274    2552 volume_manager.go:295] "The desired_state_of_world populator starts"
May 28 11:03:04.387350 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.387295    2552 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 28 11:03:04.387413 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.387402    2552 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 28 11:03:04.387467 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.387450    2552 reconstruct.go:97] "Volume reconstruction finished"
May 28 11:03:04.387467 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.387458    2552 reconciler.go:26] "Reconciler: start to sync state"
May 28 11:03:04.387849 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.387815    2552 factory.go:55] Registering systemd factory
May 28 11:03:04.387849 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.387844    2552 factory.go:223] Registration of the systemd container factory successfully
May 28 11:03:04.388063 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.388049    2552 factory.go:153] Registering CRI-O factory
May 28 11:03:04.388063 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.388065    2552 factory.go:223] Registration of the crio container factory successfully
May 28 11:03:04.388203 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.388111    2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 28 11:03:04.388203 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.388149    2552 factory.go:103] Registering Raw factory
May 28 11:03:04.388203 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.388164    2552 manager.go:1196] Started watching for new ooms in manager
May 28 11:03:04.388295 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.388232    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
May 28 11:03:04.388551 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.388539    2552 manager.go:319] Starting recovery of all containers
May 28 11:03:04.392227 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.392207    2552 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 11:03:04.397620 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.397588    2552 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-107.ec2.internal\" not found" node="ip-10-0-133-107.ec2.internal"
May 28 11:03:04.398770 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.398756    2552 manager.go:324] Recovery completed
May 28 11:03:04.402782 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.402768    2552 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 11:03:04.404663 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.404643    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientMemory"
May 28 11:03:04.404750 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.404687    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasNoDiskPressure"
May 28 11:03:04.404750 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.404702    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientPID"
May 28 11:03:04.405189 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.405174    2552 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 28 11:03:04.405189 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.405187    2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 28 11:03:04.405284 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.405205    2552 state_mem.go:36] "Initialized new in-memory state store"
May 28 11:03:04.408991 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.408980    2552 policy_none.go:49] "None policy: Start"
May 28 11:03:04.409030 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.409006    2552 memory_manager.go:186] "Starting memorymanager" policy="None"
May 28 11:03:04.409030 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.409022    2552 state_mem.go:35] "Initializing new in-memory state store"
May 28 11:03:04.451709 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.451696    2552 manager.go:341] "Starting Device Plugin manager"
May 28 11:03:04.472515 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.451750    2552 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 28 11:03:04.472515 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.451760    2552 server.go:85] "Starting device plugin registration server"
May 28 11:03:04.472515 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.451999    2552 eviction_manager.go:189] "Eviction manager: starting control loop"
May 28 11:03:04.472515 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.452009    2552 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 28 11:03:04.472515 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.452225    2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 28 11:03:04.472515 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.452328    2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 28 11:03:04.472515 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.452337    2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 28 11:03:04.472515 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.452664    2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 28 11:03:04.472515 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.452704    2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-107.ec2.internal\" not found"
May 28 11:03:04.511309 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.511278    2552 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 28 11:03:04.512507 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.512492    2552 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 28 11:03:04.512558 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.512518    2552 status_manager.go:230] "Starting to sync pod status with apiserver"
May 28 11:03:04.512558 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.512535    2552 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 28 11:03:04.512558 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.512542    2552 kubelet.go:2451] "Starting kubelet main sync loop"
May 28 11:03:04.512680 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.512580    2552 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 28 11:03:04.515541 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.515518    2552 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 11:03:04.552895 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.552845    2552 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 11:03:04.555220 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.554471    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientMemory"
May 28 11:03:04.555220 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.554504    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasNoDiskPressure"
May 28 11:03:04.555220 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.554526    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientPID"
May 28 11:03:04.555220 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.554593    2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-107.ec2.internal"
May 28 11:03:04.566699 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.566676    2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-107.ec2.internal"
May 28 11:03:04.566797 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.566700    2552 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-107.ec2.internal\": node \"ip-10-0-133-107.ec2.internal\" not found"
May 28 11:03:04.581682 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.581658    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
May 28 11:03:04.613012 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.612973    2552 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal"]
May 28 11:03:04.613101 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.613064    2552 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 11:03:04.616182 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.616168    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientMemory"
May 28 11:03:04.616255 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.616194    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasNoDiskPressure"
May 28 11:03:04.616255 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.616205    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientPID"
May 28 11:03:04.617417 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.617405    2552 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 11:03:04.617561 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.617547    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
May 28 11:03:04.617607 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.617583    2552 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 11:03:04.618068 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.618051    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientMemory"
May 28 11:03:04.618068 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.618067    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientMemory"
May 28 11:03:04.618223 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.618076    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasNoDiskPressure"
May 28 11:03:04.618223 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.618085    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasNoDiskPressure"
May 28 11:03:04.618223 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.618092    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientPID"
May 28 11:03:04.618223 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.618095    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientPID"
May 28 11:03:04.619207 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.619193    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal"
May 28 11:03:04.619269 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.619216    2552 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 11:03:04.619787 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.619773    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientMemory"
May 28 11:03:04.619857 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.619803    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasNoDiskPressure"
May 28 11:03:04.619857 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.619816    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeHasSufficientPID"
May 28 11:03:04.643658 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.643634    2552 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-107.ec2.internal\" not found" node="ip-10-0-133-107.ec2.internal"
May 28 11:03:04.647737 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.647713    2552 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-107.ec2.internal\" not found" node="ip-10-0-133-107.ec2.internal"
May 28 11:03:04.681934 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.681916    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
May 28 11:03:04.688976 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.688956    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d2ca73a2516d012ff28b4918ffe4448-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal\" (UID: \"4d2ca73a2516d012ff28b4918ffe4448\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
May 28 11:03:04.689039 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.688979    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d2ca73a2516d012ff28b4918ffe4448-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal\" (UID: \"4d2ca73a2516d012ff28b4918ffe4448\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
May 28 11:03:04.689039 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.688996    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a5e6230d598a6959a877daf9ec83d5e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-107.ec2.internal\" (UID: \"3a5e6230d598a6959a877daf9ec83d5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal"
May 28 11:03:04.782296 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.782271    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
May 28 11:03:04.789541 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.789524    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d2ca73a2516d012ff28b4918ffe4448-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal\" (UID: \"4d2ca73a2516d012ff28b4918ffe4448\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
May 28 11:03:04.789592 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.789551    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d2ca73a2516d012ff28b4918ffe4448-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal\" (UID: \"4d2ca73a2516d012ff28b4918ffe4448\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
May 28 11:03:04.789592 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.789571    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a5e6230d598a6959a877daf9ec83d5e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-107.ec2.internal\" (UID: \"3a5e6230d598a6959a877daf9ec83d5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal"
May 28 11:03:04.789656 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.789609    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a5e6230d598a6959a877daf9ec83d5e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-107.ec2.internal\" (UID: \"3a5e6230d598a6959a877daf9ec83d5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal"
May 28 11:03:04.789656 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.789637    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d2ca73a2516d012ff28b4918ffe4448-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal\" (UID: \"4d2ca73a2516d012ff28b4918ffe4448\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
May 28 11:03:04.789715 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.789637    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d2ca73a2516d012ff28b4918ffe4448-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal\" (UID: \"4d2ca73a2516d012ff28b4918ffe4448\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
May 28 11:03:04.882967 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.882899    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
May 28 11:03:04.946374 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.946339    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
May 28 11:03:04.950451 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:04.950428    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal"
May 28 11:03:04.984077 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:04.984049    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
May 28 11:03:05.084471 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:05.084435    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
May 28 11:03:05.185057 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:05.184983    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
May 28 11:03:05.285614 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:05.285585    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
May 28 11:03:05.290902 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.290882    2552 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 11:03:05.292243 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.292226    2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 28 11:03:05.292361 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.292334    2552 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 28 11:03:05.292473 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.292381    2552 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 28 11:03:05.384724 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.384693    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-27 10:58:04 +0000 UTC" deadline="2027-11-03 21:48:01.591694972 +0000 UTC"
May 28 11:03:05.384724 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.384718    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12586h44m56.206979699s"
May 28 11:03:05.385803 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:05.385778    2552 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-107.ec2.internal\" not found"
May 28 11:03:05.386614 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.386583    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 28 11:03:05.393711 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.393693    2552 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 11:03:05.396549 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.396520    2552 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 28 11:03:05.418741 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.418717    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f9lx8"
May 28 11:03:05.425864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.425844    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f9lx8"
May 28 11:03:05.431367 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:05.431337    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5e6230d598a6959a877daf9ec83d5e.slice/crio-232273ebf0c8ee7926b5d6916079fdbcc4f00c818d8f5c3152c8944ea20cb267 WatchSource:0}: Error finding container 232273ebf0c8ee7926b5d6916079fdbcc4f00c818d8f5c3152c8944ea20cb267: Status 404 returned error can't find the container with id 232273ebf0c8ee7926b5d6916079fdbcc4f00c818d8f5c3152c8944ea20cb267
May 28 11:03:05.439496 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.439481    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 11:03:05.449270 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:05.449250    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2ca73a2516d012ff28b4918ffe4448.slice/crio-a6938433e008f3074073497cd0fa40d89705934a55b3c59c6058557ce0915726 WatchSource:0}: Error finding container a6938433e008f3074073497cd0fa40d89705934a55b3c59c6058557ce0915726: Status 404 returned error can't find the container with id a6938433e008f3074073497cd0fa40d89705934a55b3c59c6058557ce0915726
May 28 11:03:05.488190 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.488164    2552 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal"
May 28 11:03:05.499720 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.499701    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 28 11:03:05.500986 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.500974    2552 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal"
May 28 11:03:05.509799 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.509782    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 28 11:03:05.515751 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.515713    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal" event={"ID":"4d2ca73a2516d012ff28b4918ffe4448","Type":"ContainerStarted","Data":"a6938433e008f3074073497cd0fa40d89705934a55b3c59c6058557ce0915726"}
May 28 11:03:05.516588 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:05.516568    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal" event={"ID":"3a5e6230d598a6959a877daf9ec83d5e","Type":"ContainerStarted","Data":"232273ebf0c8ee7926b5d6916079fdbcc4f00c818d8f5c3152c8944ea20cb267"}
May 28 11:03:06.149386 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.149227    2552 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 11:03:06.230891 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.230854    2552 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 11:03:06.360923 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.360890    2552 apiserver.go:52] "Watching apiserver"
May 28 11:03:06.368204 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.368182    2552 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 28 11:03:06.368584 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.368561    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-8xtm7","kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg","openshift-dns/node-resolver-4lh8f","openshift-image-registry/node-ca-hsvzs","openshift-multus/multus-tprqr","openshift-multus/network-metrics-daemon-5jcnw","openshift-network-diagnostics/network-check-target-kc5jm","openshift-cluster-node-tuning-operator/tuned-wzmbz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal","openshift-multus/multus-additional-cni-plugins-j8tns","openshift-network-operator/iptables-alerter-s8hjt","openshift-ovn-kubernetes/ovnkube-node-fhbhc"]
May 28 11:03:06.371040 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.371018    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:06.371164 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:06.371125    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:06.373124 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.373102    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.375117 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.375080    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 28 11:03:06.376737 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.375685    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 28 11:03:06.376737 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.375755    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4lh8f"
May 28 11:03:06.376737 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.376462    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 28 11:03:06.377832 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.377208    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ndtqs\""
May 28 11:03:06.377832 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.377739    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 28 11:03:06.378155 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.378087    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hcjq4\""
May 28 11:03:06.379391 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.379324    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 28 11:03:06.381373 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.381350    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hsvzs"
May 28 11:03:06.381448 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.381425    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tprqr"
May 28 11:03:06.383205 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.383184    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 28 11:03:06.383455 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.383439    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 28 11:03:06.383580 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.383566    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 28 11:03:06.383645 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.383630    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 28 11:03:06.383740 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.383728    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 28 11:03:06.384095 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.383858    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 28 11:03:06.384095 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.383883    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 28 11:03:06.384095 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.383949    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h44lp\""
May 28 11:03:06.384306 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.384153    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gvhqk\""
May 28 11:03:06.385596 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.385581    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8xtm7"
May 28 11:03:06.387270 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.387246    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 28 11:03:06.387486 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.387473    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r6mwn\""
May 28 11:03:06.387609 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.387593    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 28 11:03:06.387804 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.387771    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:06.387900 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:06.387858    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:06.387900 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.387872    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.390122 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.390102    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 28 11:03:06.390227 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.390189    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ftkht\""
May 28 11:03:06.390292 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.390232    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.390525 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.390505    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 28 11:03:06.391979 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.391949    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 28 11:03:06.393053 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.392141    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7bbcs\""
May 28 11:03:06.393053 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.392149    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 28 11:03:06.393053 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.392480    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s8hjt"
May 28 11:03:06.394141 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.394110    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 28 11:03:06.394248 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.394216    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 28 11:03:06.394431 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.394416    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 28 11:03:06.394504 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.394435    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h9g54\""
May 28 11:03:06.395168 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.395151    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.396618 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.396595    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-var-lib-kubelet\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.396700 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.396670    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-etc-selinux\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.396764 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.396702    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjq82\" (UniqueName: \"kubernetes.io/projected/cba5e87a-42ea-447e-8c21-439ce0bbb9fd-kube-api-access-zjq82\") pod \"node-resolver-4lh8f\" (UID: \"cba5e87a-42ea-447e-8c21-439ce0bbb9fd\") " pod="openshift-dns/node-resolver-4lh8f"
May 28 11:03:06.396764 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.396755    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgfk7\" (UniqueName: \"kubernetes.io/projected/4fd526dd-8e8c-4600-a0cb-182ce3516c6f-kube-api-access-rgfk7\") pod \"node-ca-hsvzs\" (UID: \"4fd526dd-8e8c-4600-a0cb-182ce3516c6f\") " pod="openshift-image-registry/node-ca-hsvzs"
May 28 11:03:06.396870 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.396780    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-var-lib-cni-multus\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.396870 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.396805    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-multus-daemon-config\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.396870 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.396826    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-sysctl-d\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.396870 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.396847    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-registration-dir\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.397046 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.396877    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-l6qpz\""
May 28 11:03:06.397046 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.396893    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc5tr\" (UniqueName: \"kubernetes.io/projected/a3c317fb-48c6-4e83-ad01-06395193c20c-kube-api-access-dc5tr\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.397046 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.396958    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-hostroot\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.397046 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.396984    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-etc-kubernetes\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.397046 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397010    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-sys\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.397046 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397031    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs\") pod \"network-metrics-daemon-5jcnw\" (UID: \"1f58f4ea-58e7-4a59-afd8-b41076bfa130\") " pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:06.397334 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397067    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6vnl\" (UniqueName: \"kubernetes.io/projected/1f58f4ea-58e7-4a59-afd8-b41076bfa130-kube-api-access-d6vnl\") pod \"network-metrics-daemon-5jcnw\" (UID: \"1f58f4ea-58e7-4a59-afd8-b41076bfa130\") " pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:06.397334 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397256    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 28 11:03:06.397334 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397263    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 28 11:03:06.397334 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397295    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 28 11:03:06.397334 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397270    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-device-dir\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.397558 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397371    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-var-lib-kubelet\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.397558 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397394    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-kubernetes\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.397558 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397422    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-systemd\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.397558 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397449    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-host\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.397558 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397486    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh68g\" (UniqueName: \"kubernetes.io/projected/e9125deb-cfc1-432d-acb8-31f019b7987b-kube-api-access-mh68g\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.397558 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397536    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.397558 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397491    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 28 11:03:06.397879 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397563    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cba5e87a-42ea-447e-8c21-439ce0bbb9fd-hosts-file\") pod \"node-resolver-4lh8f\" (UID: \"cba5e87a-42ea-447e-8c21-439ce0bbb9fd\") " pod="openshift-dns/node-resolver-4lh8f"
May 28 11:03:06.397879 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397566    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 28 11:03:06.397879 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397590    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51fa9d4e-2280-4b26-8d0d-c9649fa19f51-konnectivity-ca\") pod \"konnectivity-agent-8xtm7\" (UID: \"51fa9d4e-2280-4b26-8d0d-c9649fa19f51\") " pod="kube-system/konnectivity-agent-8xtm7"
May 28 11:03:06.397879 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397614    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4fd526dd-8e8c-4600-a0cb-182ce3516c6f-serviceca\") pod \"node-ca-hsvzs\" (UID: \"4fd526dd-8e8c-4600-a0cb-182ce3516c6f\") " pod="openshift-image-registry/node-ca-hsvzs"
May 28 11:03:06.397879 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397566    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 28 11:03:06.397879 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397637    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-run-netns\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.397879 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397674    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-run-multus-certs\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.397879 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397720    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxvt5\" (UniqueName: \"kubernetes.io/projected/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-kube-api-access-vxvt5\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.397879 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397759    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51fa9d4e-2280-4b26-8d0d-c9649fa19f51-agent-certs\") pod \"konnectivity-agent-8xtm7\" (UID: \"51fa9d4e-2280-4b26-8d0d-c9649fa19f51\") " pod="kube-system/konnectivity-agent-8xtm7"
May 28 11:03:06.397879 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397788    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-system-cni-dir\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.397879 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397839    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-cnibin\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.397879 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397877    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-var-lib-cni-bin\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397904    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-modprobe-d\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397938    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-lib-modules\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397964    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba5e87a-42ea-447e-8c21-439ce0bbb9fd-tmp-dir\") pod \"node-resolver-4lh8f\" (UID: \"cba5e87a-42ea-447e-8c21-439ce0bbb9fd\") " pod="openshift-dns/node-resolver-4lh8f"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.397986    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-run-k8s-cni-cncf-io\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.398002    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-multus-conf-dir\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.398017    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-tuned\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.398032    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9125deb-cfc1-432d-acb8-31f019b7987b-tmp\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.398056    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-socket-dir\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.398078    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-sys-fs\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.398101    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fd526dd-8e8c-4600-a0cb-182ce3516c6f-host\") pod \"node-ca-hsvzs\" (UID: \"4fd526dd-8e8c-4600-a0cb-182ce3516c6f\") " pod="openshift-image-registry/node-ca-hsvzs"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.398124    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-multus-cni-dir\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.398171    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-os-release\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.398208    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-cni-binary-copy\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.398246    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-multus-socket-dir-parent\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.398276    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gf74\" (UniqueName: \"kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74\") pod \"network-check-target-kc5jm\" (UID: \"c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2\") " pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.398299    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-sysconfig\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.398383 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.398325    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-sysctl-conf\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.398989 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.398346    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-run\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.426490 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.426453    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 10:58:05 +0000 UTC" deadline="2028-02-26 18:36:24.081683722 +0000 UTC"
May 28 11:03:06.426490 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.426482    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15343h33m17.655205627s"
May 28 11:03:06.488202 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.488177    2552 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 28 11:03:06.498699 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.498668    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51fa9d4e-2280-4b26-8d0d-c9649fa19f51-agent-certs\") pod \"konnectivity-agent-8xtm7\" (UID: \"51fa9d4e-2280-4b26-8d0d-c9649fa19f51\") " pod="kube-system/konnectivity-agent-8xtm7"
May 28 11:03:06.498853 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.498706    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-cnibin\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.498853 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.498729    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-var-lib-cni-bin\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.498853 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.498792    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-cnibin\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.499002 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.498853    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-lib-modules\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.499002 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.498899    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2wl4\" (UniqueName: \"kubernetes.io/projected/7546d43c-0df3-4fa0-9258-908e2c006bb8-kube-api-access-q2wl4\") pod \"iptables-alerter-s8hjt\" (UID: \"7546d43c-0df3-4fa0-9258-908e2c006bb8\") " pod="openshift-network-operator/iptables-alerter-s8hjt"
May 28 11:03:06.499002 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.498861    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-var-lib-cni-bin\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.499002 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.498953    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-run-systemd\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.499002 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.498969    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-lib-modules\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.499002 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.498985    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba5e87a-42ea-447e-8c21-439ce0bbb9fd-tmp-dir\") pod \"node-resolver-4lh8f\" (UID: \"cba5e87a-42ea-447e-8c21-439ce0bbb9fd\") " pod="openshift-dns/node-resolver-4lh8f"
May 28 11:03:06.499335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499011    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-sys-fs\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.499335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499038    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-cni-bin\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.499335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499065    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-multus-cni-dir\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.499335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499084    2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 28 11:03:06.499335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499091    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2gf74\" (UniqueName: \"kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74\") pod \"network-check-target-kc5jm\" (UID: \"c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2\") " pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:06.499335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499157    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-multus-cni-dir\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.499335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499140    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-sys-fs\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.499335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499180    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-etc-selinux\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.499335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499213    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.499335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499239    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-ovnkube-config\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.499335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499263    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-system-cni-dir\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.499335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499286    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba5e87a-42ea-447e-8c21-439ce0bbb9fd-tmp-dir\") pod \"node-resolver-4lh8f\" (UID: \"cba5e87a-42ea-447e-8c21-439ce0bbb9fd\") " pod="openshift-dns/node-resolver-4lh8f"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499291    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-os-release\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499291    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-etc-selinux\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499387    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-var-lib-kubelet\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499422    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-cni-netd\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499447    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-cni-binary-copy\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499471    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499491    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-var-lib-kubelet\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499495    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rgfk7\" (UniqueName: \"kubernetes.io/projected/4fd526dd-8e8c-4600-a0cb-182ce3516c6f-kube-api-access-rgfk7\") pod \"node-ca-hsvzs\" (UID: \"4fd526dd-8e8c-4600-a0cb-182ce3516c6f\") " pod="openshift-image-registry/node-ca-hsvzs"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499538    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-multus-daemon-config\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499564    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-sysctl-d\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499588    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-registration-dir\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499647    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dc5tr\" (UniqueName: \"kubernetes.io/projected/a3c317fb-48c6-4e83-ad01-06395193c20c-kube-api-access-dc5tr\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499675    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-run-openvswitch\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499720    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-registration-dir\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499739    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-sysctl-d\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499794    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-log-socket\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.499852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499842    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-etc-kubernetes\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499870    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-sys\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499896    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-etc-kubernetes\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499902    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs\") pod \"network-metrics-daemon-5jcnw\" (UID: \"1f58f4ea-58e7-4a59-afd8-b41076bfa130\") " pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499948    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-sys\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.499991    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-kubernetes\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:06.500012    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500025    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-systemd\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500053    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-host\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:06.500099    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs podName:1f58f4ea-58e7-4a59-afd8-b41076bfa130 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:07.000065292 +0000 UTC m=+3.039162271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs") pod "network-metrics-daemon-5jcnw" (UID: "1f58f4ea-58e7-4a59-afd8-b41076bfa130") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500112    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-host\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500122    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-var-lib-openvswitch\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500164    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-systemd\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500097    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-kubernetes\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500168    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-multus-daemon-config\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500169    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-etc-openvswitch\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500219    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.500639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500249    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4fd526dd-8e8c-4600-a0cb-182ce3516c6f-serviceca\") pod \"node-ca-hsvzs\" (UID: \"4fd526dd-8e8c-4600-a0cb-182ce3516c6f\") " pod="openshift-image-registry/node-ca-hsvzs"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500273    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-run-netns\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500293    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-run-multus-certs\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500315    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7546d43c-0df3-4fa0-9258-908e2c006bb8-iptables-alerter-script\") pod \"iptables-alerter-s8hjt\" (UID: \"7546d43c-0df3-4fa0-9258-908e2c006bb8\") " pod="openshift-network-operator/iptables-alerter-s8hjt"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500335    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-kubelet\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500335    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-run-netns\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500366    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-ovn-node-metrics-cert\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500390    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-ovnkube-script-lib\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500398    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-run-multus-certs\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500410    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pps9c\" (UniqueName: \"kubernetes.io/projected/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-kube-api-access-pps9c\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500428    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vxvt5\" (UniqueName: \"kubernetes.io/projected/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-kube-api-access-vxvt5\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500451    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-system-cni-dir\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500501    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-system-cni-dir\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500553    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-modprobe-d\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500585    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7546d43c-0df3-4fa0-9258-908e2c006bb8-host-slash\") pod \"iptables-alerter-s8hjt\" (UID: \"7546d43c-0df3-4fa0-9258-908e2c006bb8\") " pod="openshift-network-operator/iptables-alerter-s8hjt"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500622    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-env-overrides\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500643    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-modprobe-d\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.501420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500646    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-cnibin\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500651    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4fd526dd-8e8c-4600-a0cb-182ce3516c6f-serviceca\") pod \"node-ca-hsvzs\" (UID: \"4fd526dd-8e8c-4600-a0cb-182ce3516c6f\") " pod="openshift-image-registry/node-ca-hsvzs"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500674    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-run-k8s-cni-cncf-io\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500694    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-multus-conf-dir\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500710    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-tuned\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500735    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9125deb-cfc1-432d-acb8-31f019b7987b-tmp\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500750    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-run-k8s-cni-cncf-io\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500765    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-socket-dir\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500776    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-multus-conf-dir\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500791    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-slash\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500851    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-node-log\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500876    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7mmb\" (UniqueName: \"kubernetes.io/projected/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-kube-api-access-h7mmb\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500888    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-socket-dir\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500900    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fd526dd-8e8c-4600-a0cb-182ce3516c6f-host\") pod \"node-ca-hsvzs\" (UID: \"4fd526dd-8e8c-4600-a0cb-182ce3516c6f\") " pod="openshift-image-registry/node-ca-hsvzs"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500924    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-os-release\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500949    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-cni-binary-copy\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500954    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fd526dd-8e8c-4600-a0cb-182ce3516c6f-host\") pod \"node-ca-hsvzs\" (UID: \"4fd526dd-8e8c-4600-a0cb-182ce3516c6f\") " pod="openshift-image-registry/node-ca-hsvzs"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.500975    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-multus-socket-dir-parent\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.502236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501000    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-sysconfig\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501020    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-os-release\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501026    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-sysctl-conf\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501051    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-run\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501072    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-multus-socket-dir-parent\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501074    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-systemd-units\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501101    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-sysconfig\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501114    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501168    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zjq82\" (UniqueName: \"kubernetes.io/projected/cba5e87a-42ea-447e-8c21-439ce0bbb9fd-kube-api-access-zjq82\") pod \"node-resolver-4lh8f\" (UID: \"cba5e87a-42ea-447e-8c21-439ce0bbb9fd\") " pod="openshift-dns/node-resolver-4lh8f"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501174    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-run\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501197    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-var-lib-cni-multus\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501221    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-sysctl-conf\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501222    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-run-netns\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501259    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-var-lib-cni-multus\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501266    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501293    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-hostroot\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501340    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d6vnl\" (UniqueName: \"kubernetes.io/projected/1f58f4ea-58e7-4a59-afd8-b41076bfa130-kube-api-access-d6vnl\") pod \"network-metrics-daemon-5jcnw\" (UID: \"1f58f4ea-58e7-4a59-afd8-b41076bfa130\") " pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:06.502971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501367    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-device-dir\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.503672 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501379    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-hostroot\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.503672 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501392    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-run-ovn\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.503672 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501418    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-var-lib-kubelet\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.503672 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501436    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-device-dir\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.503672 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501443    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mh68g\" (UniqueName: \"kubernetes.io/projected/e9125deb-cfc1-432d-acb8-31f019b7987b-kube-api-access-mh68g\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.503672 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501478    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.503672 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501505    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cba5e87a-42ea-447e-8c21-439ce0bbb9fd-hosts-file\") pod \"node-resolver-4lh8f\" (UID: \"cba5e87a-42ea-447e-8c21-439ce0bbb9fd\") " pod="openshift-dns/node-resolver-4lh8f"
May 28 11:03:06.503672 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501530    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51fa9d4e-2280-4b26-8d0d-c9649fa19f51-konnectivity-ca\") pod \"konnectivity-agent-8xtm7\" (UID: \"51fa9d4e-2280-4b26-8d0d-c9649fa19f51\") " pod="kube-system/konnectivity-agent-8xtm7"
May 28 11:03:06.503672 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501557    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3c317fb-48c6-4e83-ad01-06395193c20c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.503672 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501565    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-cni-binary-copy\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.503672 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501606    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cba5e87a-42ea-447e-8c21-439ce0bbb9fd-hosts-file\") pod \"node-resolver-4lh8f\" (UID: \"cba5e87a-42ea-447e-8c21-439ce0bbb9fd\") " pod="openshift-dns/node-resolver-4lh8f"
May 28 11:03:06.503672 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.501646    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-host-var-lib-kubelet\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.503672 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.502043    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51fa9d4e-2280-4b26-8d0d-c9649fa19f51-konnectivity-ca\") pod \"konnectivity-agent-8xtm7\" (UID: \"51fa9d4e-2280-4b26-8d0d-c9649fa19f51\") " pod="kube-system/konnectivity-agent-8xtm7"
May 28 11:03:06.503672 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.502328    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51fa9d4e-2280-4b26-8d0d-c9649fa19f51-agent-certs\") pod \"konnectivity-agent-8xtm7\" (UID: \"51fa9d4e-2280-4b26-8d0d-c9649fa19f51\") " pod="kube-system/konnectivity-agent-8xtm7"
May 28 11:03:06.503672 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.503624    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9125deb-cfc1-432d-acb8-31f019b7987b-tmp\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.504221 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.503697    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9125deb-cfc1-432d-acb8-31f019b7987b-etc-tuned\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.504333 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:06.504318    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 11:03:06.504388 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:06.504339    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 11:03:06.504388 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:06.504351    2552 projected.go:194] Error preparing data for projected volume kube-api-access-2gf74 for pod openshift-network-diagnostics/network-check-target-kc5jm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 11:03:06.504483 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:06.504423    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74 podName:c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:07.004404802 +0000 UTC m=+3.043501787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2gf74" (UniqueName: "kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74") pod "network-check-target-kc5jm" (UID: "c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 11:03:06.506978 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.506952    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgfk7\" (UniqueName: \"kubernetes.io/projected/4fd526dd-8e8c-4600-a0cb-182ce3516c6f-kube-api-access-rgfk7\") pod \"node-ca-hsvzs\" (UID: \"4fd526dd-8e8c-4600-a0cb-182ce3516c6f\") " pod="openshift-image-registry/node-ca-hsvzs"
May 28 11:03:06.507782 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.507758    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc5tr\" (UniqueName: \"kubernetes.io/projected/a3c317fb-48c6-4e83-ad01-06395193c20c-kube-api-access-dc5tr\") pod \"aws-ebs-csi-driver-node-66nrg\" (UID: \"a3c317fb-48c6-4e83-ad01-06395193c20c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.508748 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.508723    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6vnl\" (UniqueName: \"kubernetes.io/projected/1f58f4ea-58e7-4a59-afd8-b41076bfa130-kube-api-access-d6vnl\") pod \"network-metrics-daemon-5jcnw\" (UID: \"1f58f4ea-58e7-4a59-afd8-b41076bfa130\") " pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:06.509306 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.509284    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxvt5\" (UniqueName: \"kubernetes.io/projected/0d2c92fb-bc6d-4c6e-b59a-d9c745d08344-kube-api-access-vxvt5\") pod \"multus-tprqr\" (UID: \"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344\") " pod="openshift-multus/multus-tprqr"
May 28 11:03:06.509631 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.509610    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjq82\" (UniqueName: \"kubernetes.io/projected/cba5e87a-42ea-447e-8c21-439ce0bbb9fd-kube-api-access-zjq82\") pod \"node-resolver-4lh8f\" (UID: \"cba5e87a-42ea-447e-8c21-439ce0bbb9fd\") " pod="openshift-dns/node-resolver-4lh8f"
May 28 11:03:06.510030 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.510009    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh68g\" (UniqueName: \"kubernetes.io/projected/e9125deb-cfc1-432d-acb8-31f019b7987b-kube-api-access-mh68g\") pod \"tuned-wzmbz\" (UID: \"e9125deb-cfc1-432d-acb8-31f019b7987b\") " pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.601814 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.601780    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.601814 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.601823    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-ovnkube-config\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602033 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.601849    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-system-cni-dir\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.602033 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.601870    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-os-release\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.602033 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.601895    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-cni-netd\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602033 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.601903    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602033 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.601917    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-cni-binary-copy\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.602033 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.601970    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-cni-netd\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602033 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602000    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-os-release\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.602033 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602001    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-system-cni-dir\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.602033 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602005    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602069    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-run-openvswitch\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602099    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-log-socket\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602120    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-run-openvswitch\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602123    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602166    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-var-lib-openvswitch\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602181    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-log-socket\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602196    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-etc-openvswitch\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602214    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-var-lib-openvswitch\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602225    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602238    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-etc-openvswitch\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602256    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7546d43c-0df3-4fa0-9258-908e2c006bb8-iptables-alerter-script\") pod \"iptables-alerter-s8hjt\" (UID: \"7546d43c-0df3-4fa0-9258-908e2c006bb8\") " pod="openshift-network-operator/iptables-alerter-s8hjt"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602278    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602282    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-kubelet\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602316    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-kubelet\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602326    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-ovn-node-metrics-cert\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602358    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-ovnkube-script-lib\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.602397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602385    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pps9c\" (UniqueName: \"kubernetes.io/projected/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-kube-api-access-pps9c\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602414    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7546d43c-0df3-4fa0-9258-908e2c006bb8-host-slash\") pod \"iptables-alerter-s8hjt\" (UID: \"7546d43c-0df3-4fa0-9258-908e2c006bb8\") " pod="openshift-network-operator/iptables-alerter-s8hjt"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602439    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-env-overrides\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602462    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-cnibin\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602472    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-cni-binary-copy\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602490    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-slash\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602503    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-ovnkube-config\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602523    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7546d43c-0df3-4fa0-9258-908e2c006bb8-host-slash\") pod \"iptables-alerter-s8hjt\" (UID: \"7546d43c-0df3-4fa0-9258-908e2c006bb8\") " pod="openshift-network-operator/iptables-alerter-s8hjt"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602513    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-node-log\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602542    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-node-log\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602574    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mmb\" (UniqueName: \"kubernetes.io/projected/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-kube-api-access-h7mmb\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602583    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-slash\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602617    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-systemd-units\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602643    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602679    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-run-netns\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602702    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602731    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-systemd-units\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602784    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-run-ovn\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602731    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-run-ovn\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602837    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7546d43c-0df3-4fa0-9258-908e2c006bb8-iptables-alerter-script\") pod \"iptables-alerter-s8hjt\" (UID: \"7546d43c-0df3-4fa0-9258-908e2c006bb8\") " pod="openshift-network-operator/iptables-alerter-s8hjt"
May 28 11:03:06.603864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602848    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q2wl4\" (UniqueName: \"kubernetes.io/projected/7546d43c-0df3-4fa0-9258-908e2c006bb8-kube-api-access-q2wl4\") pod \"iptables-alerter-s8hjt\" (UID: \"7546d43c-0df3-4fa0-9258-908e2c006bb8\") " pod="openshift-network-operator/iptables-alerter-s8hjt"
May 28 11:03:06.603864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602873    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-run-systemd\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602892    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-run-netns\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602911    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-cni-bin\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.602983    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-host-cni-bin\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.603020    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-run-systemd\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.603057    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-cnibin\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.603864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.603079    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-env-overrides\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.603864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.603217    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.603864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.603289    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.603864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.603379    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-ovnkube-script-lib\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.605235 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.605212    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-ovn-node-metrics-cert\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.610490 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.610411    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7mmb\" (UniqueName: \"kubernetes.io/projected/e78bfc6f-02aa-4a5c-afc7-d3a907840f53-kube-api-access-h7mmb\") pod \"multus-additional-cni-plugins-j8tns\" (UID: \"e78bfc6f-02aa-4a5c-afc7-d3a907840f53\") " pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.610621 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.610603    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2wl4\" (UniqueName: \"kubernetes.io/projected/7546d43c-0df3-4fa0-9258-908e2c006bb8-kube-api-access-q2wl4\") pod \"iptables-alerter-s8hjt\" (UID: \"7546d43c-0df3-4fa0-9258-908e2c006bb8\") " pod="openshift-network-operator/iptables-alerter-s8hjt"
May 28 11:03:06.610708 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.610690    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pps9c\" (UniqueName: \"kubernetes.io/projected/2c6961c4-e7e5-46bd-aa0d-2dc026df36e1-kube-api-access-pps9c\") pod \"ovnkube-node-fhbhc\" (UID: \"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:06.686678 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.686596    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg"
May 28 11:03:06.692591 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.692566    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4lh8f"
May 28 11:03:06.702221 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.702199    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hsvzs"
May 28 11:03:06.707745 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.707723    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tprqr"
May 28 11:03:06.714283 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.714263    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8xtm7"
May 28 11:03:06.720851 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.720831    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wzmbz"
May 28 11:03:06.729395 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.729373    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j8tns"
May 28 11:03:06.736989 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.736969    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s8hjt"
May 28 11:03:06.743663 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:06.743646    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:07.005324 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.005289    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2gf74\" (UniqueName: \"kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74\") pod \"network-check-target-kc5jm\" (UID: \"c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2\") " pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:07.005498 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.005338    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs\") pod \"network-metrics-daemon-5jcnw\" (UID: \"1f58f4ea-58e7-4a59-afd8-b41076bfa130\") " pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:07.005498 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:07.005454    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 11:03:07.005498 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:07.005483    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 11:03:07.005639 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:07.005507    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 11:03:07.005639 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:07.005515    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs podName:1f58f4ea-58e7-4a59-afd8-b41076bfa130 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:08.005496194 +0000 UTC m=+4.044593150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs") pod "network-metrics-daemon-5jcnw" (UID: "1f58f4ea-58e7-4a59-afd8-b41076bfa130") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 11:03:07.005639 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:07.005519    2552 projected.go:194] Error preparing data for projected volume kube-api-access-2gf74 for pod openshift-network-diagnostics/network-check-target-kc5jm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 11:03:07.005639 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:07.005582    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74 podName:c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:08.005566278 +0000 UTC m=+4.044663255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gf74" (UniqueName: "kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74") pod "network-check-target-kc5jm" (UID: "c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 11:03:07.061603 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:07.061569    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c317fb_48c6_4e83_ad01_06395193c20c.slice/crio-bedcf59d2ce5e3633522204b0bd664adf5a3d96f570939d57690fde98a0dc705 WatchSource:0}: Error finding container bedcf59d2ce5e3633522204b0bd664adf5a3d96f570939d57690fde98a0dc705: Status 404 returned error can't find the container with id bedcf59d2ce5e3633522204b0bd664adf5a3d96f570939d57690fde98a0dc705
May 28 11:03:07.062722 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:07.062697    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba5e87a_42ea_447e_8c21_439ce0bbb9fd.slice/crio-6c4836cd2ef1778d55652bde916409d006c47a81578f7de7b9d86b70c7377f6c WatchSource:0}: Error finding container 6c4836cd2ef1778d55652bde916409d006c47a81578f7de7b9d86b70c7377f6c: Status 404 returned error can't find the container with id 6c4836cd2ef1778d55652bde916409d006c47a81578f7de7b9d86b70c7377f6c
May 28 11:03:07.065960 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:07.065934    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd526dd_8e8c_4600_a0cb_182ce3516c6f.slice/crio-be42d5d9664c0fe3fb09c3ad596d9f344eba298d1ef6f1ca9f059ff9ecf154ef WatchSource:0}: Error finding container be42d5d9664c0fe3fb09c3ad596d9f344eba298d1ef6f1ca9f059ff9ecf154ef: Status 404 returned error can't find the container with id be42d5d9664c0fe3fb09c3ad596d9f344eba298d1ef6f1ca9f059ff9ecf154ef
May 28 11:03:07.066832 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:07.066807    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2c92fb_bc6d_4c6e_b59a_d9c745d08344.slice/crio-3cd528221d0848f2193d015a69dfb0e7018f82d76ff94e9b3f12c53412422ed1 WatchSource:0}: Error finding container 3cd528221d0848f2193d015a69dfb0e7018f82d76ff94e9b3f12c53412422ed1: Status 404 returned error can't find the container with id 3cd528221d0848f2193d015a69dfb0e7018f82d76ff94e9b3f12c53412422ed1
May 28 11:03:07.068370 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:07.068285    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6961c4_e7e5_46bd_aa0d_2dc026df36e1.slice/crio-db68a5218f53c4bbdd4f475c1b1a643ab3b84fe205fd96486113aa2e958e4d7a WatchSource:0}: Error finding container db68a5218f53c4bbdd4f475c1b1a643ab3b84fe205fd96486113aa2e958e4d7a: Status 404 returned error can't find the container with id db68a5218f53c4bbdd4f475c1b1a643ab3b84fe205fd96486113aa2e958e4d7a
May 28 11:03:07.069563 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:07.069508    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7546d43c_0df3_4fa0_9258_908e2c006bb8.slice/crio-70d462bdd6ad75c29e7226b684375b0651bddd34f767897fffaed4b4ac70ccb6 WatchSource:0}: Error finding container 70d462bdd6ad75c29e7226b684375b0651bddd34f767897fffaed4b4ac70ccb6: Status 404 returned error can't find the container with id 70d462bdd6ad75c29e7226b684375b0651bddd34f767897fffaed4b4ac70ccb6
May 28 11:03:07.070599 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:07.070257    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fa9d4e_2280_4b26_8d0d_c9649fa19f51.slice/crio-ef5b532513db5de1a85114121de7ab4b55959df308eb760bee234fd641befb4e WatchSource:0}: Error finding container ef5b532513db5de1a85114121de7ab4b55959df308eb760bee234fd641befb4e: Status 404 returned error can't find the container with id ef5b532513db5de1a85114121de7ab4b55959df308eb760bee234fd641befb4e
May 28 11:03:07.071117 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:07.071009    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9125deb_cfc1_432d_acb8_31f019b7987b.slice/crio-020fdc219c3c4bbf77b165615c8a00e358d34827f3f4153b34a9449c3ed7a2e9 WatchSource:0}: Error finding container 020fdc219c3c4bbf77b165615c8a00e358d34827f3f4153b34a9449c3ed7a2e9: Status 404 returned error can't find the container with id 020fdc219c3c4bbf77b165615c8a00e358d34827f3f4153b34a9449c3ed7a2e9
May 28 11:03:07.071747 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:07.071727    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78bfc6f_02aa_4a5c_afc7_d3a907840f53.slice/crio-2358adf1199ee8c374f4465fd1ee161e2a988434bc69f278063df032d04c3244 WatchSource:0}: Error finding container 2358adf1199ee8c374f4465fd1ee161e2a988434bc69f278063df032d04c3244: Status 404 returned error can't find the container with id 2358adf1199ee8c374f4465fd1ee161e2a988434bc69f278063df032d04c3244
May 28 11:03:07.429015 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.428952    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 10:58:05 +0000 UTC" deadline="2027-11-04 12:50:33.533580867 +0000 UTC"
May 28 11:03:07.429015 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.428987    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12601h47m26.104598315s"
May 28 11:03:07.522518 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.522482    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8tns" event={"ID":"e78bfc6f-02aa-4a5c-afc7-d3a907840f53","Type":"ContainerStarted","Data":"2358adf1199ee8c374f4465fd1ee161e2a988434bc69f278063df032d04c3244"}
May 28 11:03:07.525434 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.525394    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wzmbz" event={"ID":"e9125deb-cfc1-432d-acb8-31f019b7987b","Type":"ContainerStarted","Data":"020fdc219c3c4bbf77b165615c8a00e358d34827f3f4153b34a9449c3ed7a2e9"}
May 28 11:03:07.531549 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.531517    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s8hjt" event={"ID":"7546d43c-0df3-4fa0-9258-908e2c006bb8","Type":"ContainerStarted","Data":"70d462bdd6ad75c29e7226b684375b0651bddd34f767897fffaed4b4ac70ccb6"}
May 28 11:03:07.533645 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.533618    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tprqr" event={"ID":"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344","Type":"ContainerStarted","Data":"3cd528221d0848f2193d015a69dfb0e7018f82d76ff94e9b3f12c53412422ed1"}
May 28 11:03:07.542277 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.542249    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hsvzs" event={"ID":"4fd526dd-8e8c-4600-a0cb-182ce3516c6f","Type":"ContainerStarted","Data":"be42d5d9664c0fe3fb09c3ad596d9f344eba298d1ef6f1ca9f059ff9ecf154ef"}
May 28 11:03:07.544113 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.544088    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg" event={"ID":"a3c317fb-48c6-4e83-ad01-06395193c20c","Type":"ContainerStarted","Data":"bedcf59d2ce5e3633522204b0bd664adf5a3d96f570939d57690fde98a0dc705"}
May 28 11:03:07.546923 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.546896    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal" event={"ID":"3a5e6230d598a6959a877daf9ec83d5e","Type":"ContainerStarted","Data":"c632d5ec4312e2f7dd6fcc940c535664f580d1e728d15f99bd58d3576822333b"}
May 28 11:03:07.550686 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.550658    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8xtm7" event={"ID":"51fa9d4e-2280-4b26-8d0d-c9649fa19f51","Type":"ContainerStarted","Data":"ef5b532513db5de1a85114121de7ab4b55959df308eb760bee234fd641befb4e"}
May 28 11:03:07.555194 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.555168    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc" event={"ID":"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1","Type":"ContainerStarted","Data":"db68a5218f53c4bbdd4f475c1b1a643ab3b84fe205fd96486113aa2e958e4d7a"}
May 28 11:03:07.557641 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.557618    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4lh8f" event={"ID":"cba5e87a-42ea-447e-8c21-439ce0bbb9fd","Type":"ContainerStarted","Data":"6c4836cd2ef1778d55652bde916409d006c47a81578f7de7b9d86b70c7377f6c"}
May 28 11:03:07.565351 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.565321    2552 generic.go:358] "Generic (PLEG): container finished" podID="4d2ca73a2516d012ff28b4918ffe4448" containerID="db747112504b76436bbd6e6b61965c24544ce94f7c1fbcb413d096c087901bb7" exitCode=0
May 28 11:03:07.565467 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.565366    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal" event={"ID":"4d2ca73a2516d012ff28b4918ffe4448","Type":"ContainerDied","Data":"db747112504b76436bbd6e6b61965c24544ce94f7c1fbcb413d096c087901bb7"}
May 28 11:03:07.587736 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:07.587669    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-107.ec2.internal" podStartSLOduration=2.58765429 podStartE2EDuration="2.58765429s" podCreationTimestamp="2026-05-28 11:03:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 11:03:07.560528793 +0000 UTC m=+3.599625753" watchObservedRunningTime="2026-05-28 11:03:07.58765429 +0000 UTC m=+3.626751269"
May 28 11:03:08.013798 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:08.013759    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2gf74\" (UniqueName: \"kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74\") pod \"network-check-target-kc5jm\" (UID: \"c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2\") " pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:08.013986 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:08.013839    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs\") pod \"network-metrics-daemon-5jcnw\" (UID: \"1f58f4ea-58e7-4a59-afd8-b41076bfa130\") " pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:08.014050 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:08.013983    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 11:03:08.014050 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:08.014045    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs podName:1f58f4ea-58e7-4a59-afd8-b41076bfa130 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:10.014025903 +0000 UTC m=+6.053122873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs") pod "network-metrics-daemon-5jcnw" (UID: "1f58f4ea-58e7-4a59-afd8-b41076bfa130") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 11:03:08.018171 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:08.014460    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 11:03:08.018171 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:08.014480    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 11:03:08.018171 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:08.014492    2552 projected.go:194] Error preparing data for projected volume kube-api-access-2gf74 for pod openshift-network-diagnostics/network-check-target-kc5jm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 11:03:08.018171 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:08.014537    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74 podName:c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:10.014520727 +0000 UTC m=+6.053617702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gf74" (UniqueName: "kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74") pod "network-check-target-kc5jm" (UID: "c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 11:03:08.514325 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:08.514293    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:08.514777 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:08.514420    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:08.514961 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:08.514941    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:08.515074 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:08.515053    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:08.579866 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:08.579827    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal" event={"ID":"4d2ca73a2516d012ff28b4918ffe4448","Type":"ContainerStarted","Data":"1479c69857be4243076ac24371f6521b9909a47c6d1c4cd3bb9cc75d27f27c0f"}
May 28 11:03:10.030045 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:10.030003    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2gf74\" (UniqueName: \"kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74\") pod \"network-check-target-kc5jm\" (UID: \"c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2\") " pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:10.030540 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:10.030070    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs\") pod \"network-metrics-daemon-5jcnw\" (UID: \"1f58f4ea-58e7-4a59-afd8-b41076bfa130\") " pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:10.030540 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:10.030236    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 11:03:10.030540 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:10.030302    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs podName:1f58f4ea-58e7-4a59-afd8-b41076bfa130 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:14.030283686 +0000 UTC m=+10.069380650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs") pod "network-metrics-daemon-5jcnw" (UID: "1f58f4ea-58e7-4a59-afd8-b41076bfa130") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 11:03:10.030841 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:10.030706    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 11:03:10.030841 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:10.030730    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 11:03:10.030841 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:10.030743    2552 projected.go:194] Error preparing data for projected volume kube-api-access-2gf74 for pod openshift-network-diagnostics/network-check-target-kc5jm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 11:03:10.030841 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:10.030787    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74 podName:c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:14.030771977 +0000 UTC m=+10.069868947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gf74" (UniqueName: "kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74") pod "network-check-target-kc5jm" (UID: "c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 11:03:10.515424 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:10.515389    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:10.515608 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:10.515520    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:10.515984 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:10.515954    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:10.516090 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:10.516056    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:12.514274 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:12.513708    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:12.514274 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:12.513878    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:12.514274 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:12.513941    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:12.514274 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:12.514064    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:14.061919 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:14.061682    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs\") pod \"network-metrics-daemon-5jcnw\" (UID: \"1f58f4ea-58e7-4a59-afd8-b41076bfa130\") " pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:14.061919 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:14.061756    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2gf74\" (UniqueName: \"kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74\") pod \"network-check-target-kc5jm\" (UID: \"c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2\") " pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:14.061919 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:14.061840    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 11:03:14.061919 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:14.061853    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 11:03:14.061919 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:14.061867    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 11:03:14.061919 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:14.061878    2552 projected.go:194] Error preparing data for projected volume kube-api-access-2gf74 for pod openshift-network-diagnostics/network-check-target-kc5jm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 11:03:14.061919 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:14.061902    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs podName:1f58f4ea-58e7-4a59-afd8-b41076bfa130 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:22.061883288 +0000 UTC m=+18.100980254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs") pod "network-metrics-daemon-5jcnw" (UID: "1f58f4ea-58e7-4a59-afd8-b41076bfa130") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 11:03:14.061919 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:14.061919    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74 podName:c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:22.061911356 +0000 UTC m=+18.101008323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gf74" (UniqueName: "kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74") pod "network-check-target-kc5jm" (UID: "c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 11:03:14.514623 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:14.514287    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:14.514623 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:14.514396    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:14.514623 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:14.514461    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:14.514623 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:14.514568    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:16.512933 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:16.512895    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:16.513315 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:16.512897    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:16.513315 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:16.513031    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:16.513315 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:16.513085    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:18.513477 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:18.513418    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:18.513477 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:18.513459    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:18.513966 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:18.513555    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:18.513966 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:18.513707    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:20.513746 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:20.513710    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:20.514334 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:20.513709    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:20.514334 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:20.513822    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:20.514334 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:20.513924    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:22.119727 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:22.119686    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2gf74\" (UniqueName: \"kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74\") pod \"network-check-target-kc5jm\" (UID: \"c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2\") " pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:22.119727 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:22.119737    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs\") pod \"network-metrics-daemon-5jcnw\" (UID: \"1f58f4ea-58e7-4a59-afd8-b41076bfa130\") " pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:22.120263 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:22.119846    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 11:03:22.120263 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:22.119873    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 11:03:22.120263 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:22.119903    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 11:03:22.120263 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:22.119914    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs podName:1f58f4ea-58e7-4a59-afd8-b41076bfa130 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:38.119894282 +0000 UTC m=+34.158991325 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs") pod "network-metrics-daemon-5jcnw" (UID: "1f58f4ea-58e7-4a59-afd8-b41076bfa130") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 11:03:22.120263 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:22.119916    2552 projected.go:194] Error preparing data for projected volume kube-api-access-2gf74 for pod openshift-network-diagnostics/network-check-target-kc5jm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 11:03:22.120263 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:22.119983    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74 podName:c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:38.119966105 +0000 UTC m=+34.159063071 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gf74" (UniqueName: "kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74") pod "network-check-target-kc5jm" (UID: "c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 11:03:22.512849 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:22.512783    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:22.512849 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:22.512783    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:22.513081 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:22.512904    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:22.513157 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:22.513078    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:24.513879 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:24.513834    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:24.514358 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:24.513963    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:24.514358 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:24.514048    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:24.514358 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:24.514161    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:25.610062 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.609907    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8xtm7" event={"ID":"51fa9d4e-2280-4b26-8d0d-c9649fa19f51","Type":"ContainerStarted","Data":"23fbac401a0cd9076ead22269395d86c03cb7b32f4e0d88ce43e1146b4694a9e"}
May 28 11:03:25.611859 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.611838    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc" event={"ID":"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1","Type":"ContainerStarted","Data":"78ea7162679c47e550bc234df19e807255102c9da01dc37f9752a9929d1f18b7"}
May 28 11:03:25.611943 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.611867    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc" event={"ID":"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1","Type":"ContainerStarted","Data":"f0684504e9078b6cf4e3f7e9fc33ed883582be5ca0139a0e0b3915e4e42a0fd0"}
May 28 11:03:25.611943 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.611882    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc" event={"ID":"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1","Type":"ContainerStarted","Data":"a9d0ca434c14c826505982002c541de07f3c915e8cd4cd7984e7f1d85b769326"}
May 28 11:03:25.611943 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.611891    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc" event={"ID":"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1","Type":"ContainerStarted","Data":"7c9bed171213dc2542531d921bd8051b70eda9361e2497d1d8af836b010c3637"}
May 28 11:03:25.612994 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.612971    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4lh8f" event={"ID":"cba5e87a-42ea-447e-8c21-439ce0bbb9fd","Type":"ContainerStarted","Data":"47b446b90a0da118bb8647a5de9bfbc1a13fd15590559ddc8cd75527b16310bd"}
May 28 11:03:25.614186 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.614165    2552 generic.go:358] "Generic (PLEG): container finished" podID="e78bfc6f-02aa-4a5c-afc7-d3a907840f53" containerID="2b82d7cec90c4b551d3129394a426108d9f13b79b82565dd2a10f65958437f4c" exitCode=0
May 28 11:03:25.614261 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.614224    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8tns" event={"ID":"e78bfc6f-02aa-4a5c-afc7-d3a907840f53","Type":"ContainerDied","Data":"2b82d7cec90c4b551d3129394a426108d9f13b79b82565dd2a10f65958437f4c"}
May 28 11:03:25.615577 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.615425    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wzmbz" event={"ID":"e9125deb-cfc1-432d-acb8-31f019b7987b","Type":"ContainerStarted","Data":"43331ca7b622403c269031a04ad99c8c707fb712945567834850ba3c2d050a1c"}
May 28 11:03:25.616750 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.616726    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tprqr" event={"ID":"0d2c92fb-bc6d-4c6e-b59a-d9c745d08344","Type":"ContainerStarted","Data":"ce8bff7ed5e9073c76c263b9d380936dfe31424dc08778b03df6f6108741f05f"}
May 28 11:03:25.617984 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.617962    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hsvzs" event={"ID":"4fd526dd-8e8c-4600-a0cb-182ce3516c6f","Type":"ContainerStarted","Data":"78b0c9f57c05be5ce7b06f218ebc0769505326365812e4f369a5649cb167e613"}
May 28 11:03:25.619036 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.619018    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg" event={"ID":"a3c317fb-48c6-4e83-ad01-06395193c20c","Type":"ContainerStarted","Data":"a411c653a15b638a51acd3fcb61f502e37d6fbdc874bf278e3df2da4234031ba"}
May 28 11:03:25.624304 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.624269    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-107.ec2.internal" podStartSLOduration=20.624259638 podStartE2EDuration="20.624259638s" podCreationTimestamp="2026-05-28 11:03:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 11:03:08.595326472 +0000 UTC m=+4.634423448" watchObservedRunningTime="2026-05-28 11:03:25.624259638 +0000 UTC m=+21.663356618"
May 28 11:03:25.635999 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.635952    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8xtm7" podStartSLOduration=3.683133486 podStartE2EDuration="21.635939015s" podCreationTimestamp="2026-05-28 11:03:04 +0000 UTC" firstStartedPulling="2026-05-28 11:03:07.097105344 +0000 UTC m=+3.136202304" lastFinishedPulling="2026-05-28 11:03:25.049910872 +0000 UTC m=+21.089007833" observedRunningTime="2026-05-28 11:03:25.623853644 +0000 UTC m=+21.662950623" watchObservedRunningTime="2026-05-28 11:03:25.635939015 +0000 UTC m=+21.675035994"
May 28 11:03:25.636217 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.636187    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-4lh8f" podStartSLOduration=3.650947934 podStartE2EDuration="21.636178206s" podCreationTimestamp="2026-05-28 11:03:04 +0000 UTC" firstStartedPulling="2026-05-28 11:03:07.064669838 +0000 UTC m=+3.103766801" lastFinishedPulling="2026-05-28 11:03:25.049900116 +0000 UTC m=+21.088997073" observedRunningTime="2026-05-28 11:03:25.635818305 +0000 UTC m=+21.674915283" watchObservedRunningTime="2026-05-28 11:03:25.636178206 +0000 UTC m=+21.675275185"
May 28 11:03:25.666238 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.666202    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hsvzs" podStartSLOduration=11.753856617 podStartE2EDuration="21.666189693s" podCreationTimestamp="2026-05-28 11:03:04 +0000 UTC" firstStartedPulling="2026-05-28 11:03:07.067721143 +0000 UTC m=+3.106818116" lastFinishedPulling="2026-05-28 11:03:16.98005422 +0000 UTC m=+13.019151192" observedRunningTime="2026-05-28 11:03:25.648666583 +0000 UTC m=+21.687763562" watchObservedRunningTime="2026-05-28 11:03:25.666189693 +0000 UTC m=+21.705286666"
May 28 11:03:25.666328 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.666265    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-tprqr" podStartSLOduration=3.672870897 podStartE2EDuration="21.666261888s" podCreationTimestamp="2026-05-28 11:03:04 +0000 UTC" firstStartedPulling="2026-05-28 11:03:07.068633434 +0000 UTC m=+3.107730395" lastFinishedPulling="2026-05-28 11:03:25.062024416 +0000 UTC m=+21.101121386" observedRunningTime="2026-05-28 11:03:25.665797634 +0000 UTC m=+21.704894613" watchObservedRunningTime="2026-05-28 11:03:25.666261888 +0000 UTC m=+21.705358868"
May 28 11:03:25.703469 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:25.703434    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wzmbz" podStartSLOduration=3.748450493 podStartE2EDuration="21.703421653s" podCreationTimestamp="2026-05-28 11:03:04 +0000 UTC" firstStartedPulling="2026-05-28 11:03:07.097084903 +0000 UTC m=+3.136181861" lastFinishedPulling="2026-05-28 11:03:25.052056049 +0000 UTC m=+21.091153021" observedRunningTime="2026-05-28 11:03:25.684270018 +0000 UTC m=+21.723366981" watchObservedRunningTime="2026-05-28 11:03:25.703421653 +0000 UTC m=+21.742518633"
May 28 11:03:26.513988 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:26.513959    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:26.514220 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:26.514056    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:26.514220 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:26.514115    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:26.514374 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:26.514236    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:26.599843 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:26.599647    2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 28 11:03:26.622207 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:26.622176    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s8hjt" event={"ID":"7546d43c-0df3-4fa0-9258-908e2c006bb8","Type":"ContainerStarted","Data":"61463b4ecb43022499804a996c9f37967bf7751a7a0ea80c7e9a5f25f221f11f"}
May 28 11:03:26.624019 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:26.623987    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg" event={"ID":"a3c317fb-48c6-4e83-ad01-06395193c20c","Type":"ContainerStarted","Data":"02e6d294d677125bc1a76cc552fd6c482f5d7b42ba6793fb9c358ac79d335f33"}
May 28 11:03:26.626802 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:26.626778    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc" event={"ID":"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1","Type":"ContainerStarted","Data":"8b8e9ffac69e763a9c344d9d98dc92dac1cd4dbe5bb02e9f055710b9219adee4"}
May 28 11:03:26.626914 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:26.626813    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc" event={"ID":"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1","Type":"ContainerStarted","Data":"9faa1fd9fff692553bda3c4b41fbfafe3f38cb4b3ca7ae41e91c3f1d91637c1b"}
May 28 11:03:26.635265 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:26.635217    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s8hjt" podStartSLOduration=4.7029756240000005 podStartE2EDuration="22.63520337s" podCreationTimestamp="2026-05-28 11:03:04 +0000 UTC" firstStartedPulling="2026-05-28 11:03:07.097096769 +0000 UTC m=+3.136193741" lastFinishedPulling="2026-05-28 11:03:25.029324526 +0000 UTC m=+21.068421487" observedRunningTime="2026-05-28 11:03:26.635173231 +0000 UTC m=+22.674270211" watchObservedRunningTime="2026-05-28 11:03:26.63520337 +0000 UTC m=+22.674300349"
May 28 11:03:27.464823 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:27.464661    2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-28T11:03:26.599838732Z","UUID":"c4712352-4b25-4398-8fc6-688532250dea","Handler":null,"Name":"","Endpoint":""}
May 28 11:03:27.466361 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:27.466336    2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 28 11:03:27.466361 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:27.466364    2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 28 11:03:28.513519 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:28.513464    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:28.513519 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:28.513499    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:28.514115 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:28.513598    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:28.514115 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:28.513745    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:28.592916 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:28.592881    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8xtm7"
May 28 11:03:28.631849 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:28.631816    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg" event={"ID":"a3c317fb-48c6-4e83-ad01-06395193c20c","Type":"ContainerStarted","Data":"42935f0d67969f87d35bdd639693caced09ebf9aebc390f20ae3d8ef54630bba"}
May 28 11:03:28.635058 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:28.635034    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc" event={"ID":"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1","Type":"ContainerStarted","Data":"37c15ece33ab646f337394a35ed579f1ad70b5494c0529691fb8c78dcd607f45"}
May 28 11:03:28.648564 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:28.648521    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66nrg" podStartSLOduration=3.9893927160000002 podStartE2EDuration="24.648506193s" podCreationTimestamp="2026-05-28 11:03:04 +0000 UTC" firstStartedPulling="2026-05-28 11:03:07.063941082 +0000 UTC m=+3.103038041" lastFinishedPulling="2026-05-28 11:03:27.723054561 +0000 UTC m=+23.762151518" observedRunningTime="2026-05-28 11:03:28.648329885 +0000 UTC m=+24.687426884" watchObservedRunningTime="2026-05-28 11:03:28.648506193 +0000 UTC m=+24.687603171"
May 28 11:03:30.018302 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:30.018276    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8xtm7"
May 28 11:03:30.019447 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:30.019430    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8xtm7"
May 28 11:03:30.512983 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:30.512953    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:30.512983 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:30.512981    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:30.513231 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:30.513104    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:30.513271 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:30.513238    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:30.639398 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:30.639365    2552 generic.go:358] "Generic (PLEG): container finished" podID="e78bfc6f-02aa-4a5c-afc7-d3a907840f53" containerID="39b502c5fa314fee6e496be81229dc35e21689e81a272aa2862fbc6d461848d2" exitCode=0
May 28 11:03:30.639562 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:30.639447    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8tns" event={"ID":"e78bfc6f-02aa-4a5c-afc7-d3a907840f53","Type":"ContainerDied","Data":"39b502c5fa314fee6e496be81229dc35e21689e81a272aa2862fbc6d461848d2"}
May 28 11:03:30.644185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:30.644164    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc" event={"ID":"2c6961c4-e7e5-46bd-aa0d-2dc026df36e1","Type":"ContainerStarted","Data":"489764a5cefea978060db3845559017ba3c815fd3e8ef40be340e012fa23cd6a"}
May 28 11:03:30.644929 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:30.644458    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:30.645035 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:30.644985    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8xtm7"
May 28 11:03:30.659370 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:30.659350    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:30.682269 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:30.682229    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc" podStartSLOduration=8.506073834 podStartE2EDuration="26.682217884s" podCreationTimestamp="2026-05-28 11:03:04 +0000 UTC" firstStartedPulling="2026-05-28 11:03:07.070214873 +0000 UTC m=+3.109311834" lastFinishedPulling="2026-05-28 11:03:25.246358913 +0000 UTC m=+21.285455884" observedRunningTime="2026-05-28 11:03:30.681679144 +0000 UTC m=+26.720776123" watchObservedRunningTime="2026-05-28 11:03:30.682217884 +0000 UTC m=+26.721314863"
May 28 11:03:31.197072 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:31.197036    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:31.647045 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:31.647018    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:31.665787 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:31.665749    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:03:31.943550 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:31.943471    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5jcnw"]
May 28 11:03:31.943710 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:31.943594    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:31.943773 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:31.943703    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:31.946341 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:31.946319    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kc5jm"]
May 28 11:03:31.946442 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:31.946407    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:31.946501 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:31.946468    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:32.650328 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:32.650287    2552 generic.go:358] "Generic (PLEG): container finished" podID="e78bfc6f-02aa-4a5c-afc7-d3a907840f53" containerID="cc573bb24edca5c548ca6b4a558efd285e23e6dbf228f8ddcef839a77ebb28c1" exitCode=0
May 28 11:03:32.650714 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:32.650379    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8tns" event={"ID":"e78bfc6f-02aa-4a5c-afc7-d3a907840f53","Type":"ContainerDied","Data":"cc573bb24edca5c548ca6b4a558efd285e23e6dbf228f8ddcef839a77ebb28c1"}
May 28 11:03:33.513754 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:33.513715    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:33.513924 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:33.513716    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:33.513924 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:33.513816    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:33.513924 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:33.513878    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:34.656316 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:34.656280    2552 generic.go:358] "Generic (PLEG): container finished" podID="e78bfc6f-02aa-4a5c-afc7-d3a907840f53" containerID="f4a0d9838b4f097c93ba14a90fff986da783a1db5cd88b2a709efe1960230979" exitCode=0
May 28 11:03:34.656944 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:34.656320    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8tns" event={"ID":"e78bfc6f-02aa-4a5c-afc7-d3a907840f53","Type":"ContainerDied","Data":"f4a0d9838b4f097c93ba14a90fff986da783a1db5cd88b2a709efe1960230979"}
May 28 11:03:35.512992 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:35.512956    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:35.513200 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:35.512956    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:35.513200 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:35.513116    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:35.513319 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:35.513196    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:37.513078 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.513045    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:37.513628 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.513047    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:37.513628 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:37.513179    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcnw" podUID="1f58f4ea-58e7-4a59-afd8-b41076bfa130"
May 28 11:03:37.513628 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:37.513250    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kc5jm" podUID="c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2"
May 28 11:03:37.801339 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.801260    2552 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-107.ec2.internal" event="NodeReady"
May 28 11:03:37.801491 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.801408    2552 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 28 11:03:37.843907 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.843873    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7c95j"]
May 28 11:03:37.846000 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.845975    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7c95j"
May 28 11:03:37.846883 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.846860    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8pg7p"]
May 28 11:03:37.848000 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.847971    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 28 11:03:37.848236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.847979    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 28 11:03:37.848331 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.848276    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nq6bj\""
May 28 11:03:37.848645 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.848624    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:03:37.850759 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.850740    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 28 11:03:37.850869 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.850797    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 28 11:03:37.850869 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.850810    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 28 11:03:37.851114 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.851089    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8gjbt\""
May 28 11:03:37.856912 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.855998    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7c95j"]
May 28 11:03:37.861381 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.861359    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8pg7p"]
May 28 11:03:37.941152 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.941101    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcv9v\" (UniqueName: \"kubernetes.io/projected/99ba1953-8c33-4ce7-81dd-ff524369264f-kube-api-access-dcv9v\") pod \"ingress-canary-8pg7p\" (UID: \"99ba1953-8c33-4ce7-81dd-ff524369264f\") " pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:03:37.941310 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.941163    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/566c95b3-6db1-4296-8ec0-f823019d6a40-config-volume\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:37.941310 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.941195    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:37.941310 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.941216    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmsxc\" (UniqueName: \"kubernetes.io/projected/566c95b3-6db1-4296-8ec0-f823019d6a40-kube-api-access-zmsxc\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:37.941507 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.941318    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566c95b3-6db1-4296-8ec0-f823019d6a40-tmp-dir\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:37.941507 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:37.941399    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert\") pod \"ingress-canary-8pg7p\" (UID: \"99ba1953-8c33-4ce7-81dd-ff524369264f\") " pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:03:38.042467 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:38.042429    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566c95b3-6db1-4296-8ec0-f823019d6a40-tmp-dir\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:38.042628 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:38.042475    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert\") pod \"ingress-canary-8pg7p\" (UID: \"99ba1953-8c33-4ce7-81dd-ff524369264f\") " pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:03:38.042628 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:38.042521    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dcv9v\" (UniqueName: \"kubernetes.io/projected/99ba1953-8c33-4ce7-81dd-ff524369264f-kube-api-access-dcv9v\") pod \"ingress-canary-8pg7p\" (UID: \"99ba1953-8c33-4ce7-81dd-ff524369264f\") " pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:03:38.042628 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:38.042543    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/566c95b3-6db1-4296-8ec0-f823019d6a40-config-volume\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:38.042628 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:38.042566    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:38.042628 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:38.042588    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsxc\" (UniqueName: \"kubernetes.io/projected/566c95b3-6db1-4296-8ec0-f823019d6a40-kube-api-access-zmsxc\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:38.042878 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:38.042696    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 11:03:38.042878 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:38.042777    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert podName:99ba1953-8c33-4ce7-81dd-ff524369264f nodeName:}" failed. No retries permitted until 2026-05-28 11:03:38.542757146 +0000 UTC m=+34.581854119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert") pod "ingress-canary-8pg7p" (UID: "99ba1953-8c33-4ce7-81dd-ff524369264f") : secret "canary-serving-cert" not found
May 28 11:03:38.042878 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:38.042804    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566c95b3-6db1-4296-8ec0-f823019d6a40-tmp-dir\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:38.043035 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:38.042885    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 11:03:38.043035 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:38.042949    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls podName:566c95b3-6db1-4296-8ec0-f823019d6a40 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:38.542931853 +0000 UTC m=+34.582028809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls") pod "dns-default-7c95j" (UID: "566c95b3-6db1-4296-8ec0-f823019d6a40") : secret "dns-default-metrics-tls" not found
May 28 11:03:38.043342 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:38.043319    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/566c95b3-6db1-4296-8ec0-f823019d6a40-config-volume\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:38.056326 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:38.056107    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmsxc\" (UniqueName: \"kubernetes.io/projected/566c95b3-6db1-4296-8ec0-f823019d6a40-kube-api-access-zmsxc\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:38.056326 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:38.056182    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcv9v\" (UniqueName: \"kubernetes.io/projected/99ba1953-8c33-4ce7-81dd-ff524369264f-kube-api-access-dcv9v\") pod \"ingress-canary-8pg7p\" (UID: \"99ba1953-8c33-4ce7-81dd-ff524369264f\") " pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:03:38.143180 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:38.143125    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs\") pod \"network-metrics-daemon-5jcnw\" (UID: \"1f58f4ea-58e7-4a59-afd8-b41076bfa130\") " pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:38.143334 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:38.143234    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2gf74\" (UniqueName: \"kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74\") pod \"network-check-target-kc5jm\" (UID: \"c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2\") " pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:38.143334 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:38.143287    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 11:03:38.143437 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:38.143369    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs podName:1f58f4ea-58e7-4a59-afd8-b41076bfa130 nodeName:}" failed. No retries permitted until 2026-05-28 11:04:10.143353425 +0000 UTC m=+66.182450401 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs") pod "network-metrics-daemon-5jcnw" (UID: "1f58f4ea-58e7-4a59-afd8-b41076bfa130") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 11:03:38.143437 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:38.143378    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 11:03:38.143437 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:38.143394    2552 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 11:03:38.143437 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:38.143405    2552 projected.go:194] Error preparing data for projected volume kube-api-access-2gf74 for pod openshift-network-diagnostics/network-check-target-kc5jm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 11:03:38.143599 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:38.143459    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74 podName:c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2 nodeName:}" failed. No retries permitted until 2026-05-28 11:04:10.143447162 +0000 UTC m=+66.182544118 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2gf74" (UniqueName: "kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74") pod "network-check-target-kc5jm" (UID: "c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 11:03:38.546563 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:38.546524    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:38.547026 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:38.546607    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert\") pod \"ingress-canary-8pg7p\" (UID: \"99ba1953-8c33-4ce7-81dd-ff524369264f\") " pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:03:38.547026 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:38.546732    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 11:03:38.547026 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:38.546794    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls podName:566c95b3-6db1-4296-8ec0-f823019d6a40 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:39.546780958 +0000 UTC m=+35.585877920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls") pod "dns-default-7c95j" (UID: "566c95b3-6db1-4296-8ec0-f823019d6a40") : secret "dns-default-metrics-tls" not found
May 28 11:03:38.547026 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:38.546730    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 11:03:38.547026 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:38.546894    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert podName:99ba1953-8c33-4ce7-81dd-ff524369264f nodeName:}" failed. No retries permitted until 2026-05-28 11:03:39.546875545 +0000 UTC m=+35.585972514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert") pod "ingress-canary-8pg7p" (UID: "99ba1953-8c33-4ce7-81dd-ff524369264f") : secret "canary-serving-cert" not found
May 28 11:03:39.513451 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:39.513407    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:03:39.513634 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:39.513407    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:03:39.515912 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:39.515886    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 28 11:03:39.515912 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:39.515907    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 28 11:03:39.516357 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:39.516338    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 28 11:03:39.516357 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:39.516353    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nncjx\""
May 28 11:03:39.516456 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:39.516339    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7rhrk\""
May 28 11:03:39.554091 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:39.554059    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:39.554487 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:39.554166    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert\") pod \"ingress-canary-8pg7p\" (UID: \"99ba1953-8c33-4ce7-81dd-ff524369264f\") " pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:03:39.554487 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:39.554252    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 11:03:39.554487 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:39.554274    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 11:03:39.554487 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:39.554322    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls podName:566c95b3-6db1-4296-8ec0-f823019d6a40 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:41.554301633 +0000 UTC m=+37.593398607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls") pod "dns-default-7c95j" (UID: "566c95b3-6db1-4296-8ec0-f823019d6a40") : secret "dns-default-metrics-tls" not found
May 28 11:03:39.554487 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:39.554343    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert podName:99ba1953-8c33-4ce7-81dd-ff524369264f nodeName:}" failed. No retries permitted until 2026-05-28 11:03:41.554332838 +0000 UTC m=+37.593429795 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert") pod "ingress-canary-8pg7p" (UID: "99ba1953-8c33-4ce7-81dd-ff524369264f") : secret "canary-serving-cert" not found
May 28 11:03:40.672547 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:40.672510    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8tns" event={"ID":"e78bfc6f-02aa-4a5c-afc7-d3a907840f53","Type":"ContainerStarted","Data":"620eb9049b28fca189e94a04a39948f7dfeaae1d3bceac48b1640afa60ce958e"}
May 28 11:03:41.568185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:41.568149    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert\") pod \"ingress-canary-8pg7p\" (UID: \"99ba1953-8c33-4ce7-81dd-ff524369264f\") " pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:03:41.568319 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:41.568206    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:41.568319 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:41.568304    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 11:03:41.568385 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:41.568367    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls podName:566c95b3-6db1-4296-8ec0-f823019d6a40 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:45.568352304 +0000 UTC m=+41.607449261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls") pod "dns-default-7c95j" (UID: "566c95b3-6db1-4296-8ec0-f823019d6a40") : secret "dns-default-metrics-tls" not found
May 28 11:03:41.568462 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:41.568304    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 11:03:41.568462 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:41.568435    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert podName:99ba1953-8c33-4ce7-81dd-ff524369264f nodeName:}" failed. No retries permitted until 2026-05-28 11:03:45.568423642 +0000 UTC m=+41.607520609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert") pod "ingress-canary-8pg7p" (UID: "99ba1953-8c33-4ce7-81dd-ff524369264f") : secret "canary-serving-cert" not found
May 28 11:03:41.679178 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:41.679120    2552 generic.go:358] "Generic (PLEG): container finished" podID="e78bfc6f-02aa-4a5c-afc7-d3a907840f53" containerID="620eb9049b28fca189e94a04a39948f7dfeaae1d3bceac48b1640afa60ce958e" exitCode=0
May 28 11:03:41.679178 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:41.679158    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8tns" event={"ID":"e78bfc6f-02aa-4a5c-afc7-d3a907840f53","Type":"ContainerDied","Data":"620eb9049b28fca189e94a04a39948f7dfeaae1d3bceac48b1640afa60ce958e"}
May 28 11:03:42.683465 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:42.683430    2552 generic.go:358] "Generic (PLEG): container finished" podID="e78bfc6f-02aa-4a5c-afc7-d3a907840f53" containerID="1d4c280fee37858ed3c8c5e63a74194ff94a0d9112c0c39f267862970b2089f6" exitCode=0
May 28 11:03:42.683830 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:42.683489    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8tns" event={"ID":"e78bfc6f-02aa-4a5c-afc7-d3a907840f53","Type":"ContainerDied","Data":"1d4c280fee37858ed3c8c5e63a74194ff94a0d9112c0c39f267862970b2089f6"}
May 28 11:03:43.687798 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:43.687764    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8tns" event={"ID":"e78bfc6f-02aa-4a5c-afc7-d3a907840f53","Type":"ContainerStarted","Data":"b4dfe9c7be5b4d7235c9a53aa953fbffec36a6e2c18686e3ceacd9aba7463ec1"}
May 28 11:03:43.712769 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:43.712720    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j8tns" podStartSLOduration=6.351746106 podStartE2EDuration="39.712704805s" podCreationTimestamp="2026-05-28 11:03:04 +0000 UTC" firstStartedPulling="2026-05-28 11:03:07.097162252 +0000 UTC m=+3.136259209" lastFinishedPulling="2026-05-28 11:03:40.458120951 +0000 UTC m=+36.497217908" observedRunningTime="2026-05-28 11:03:43.712284265 +0000 UTC m=+39.751381245" watchObservedRunningTime="2026-05-28 11:03:43.712704805 +0000 UTC m=+39.751801784"
May 28 11:03:45.595813 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:45.595769    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:45.596240 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:45.595918    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert\") pod \"ingress-canary-8pg7p\" (UID: \"99ba1953-8c33-4ce7-81dd-ff524369264f\") " pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:03:45.596240 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:45.595926    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 11:03:45.596240 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:45.595998    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls podName:566c95b3-6db1-4296-8ec0-f823019d6a40 nodeName:}" failed. No retries permitted until 2026-05-28 11:03:53.595982233 +0000 UTC m=+49.635079196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls") pod "dns-default-7c95j" (UID: "566c95b3-6db1-4296-8ec0-f823019d6a40") : secret "dns-default-metrics-tls" not found
May 28 11:03:45.596240 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:45.596002    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 11:03:45.596240 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:45.596048    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert podName:99ba1953-8c33-4ce7-81dd-ff524369264f nodeName:}" failed. No retries permitted until 2026-05-28 11:03:53.596037223 +0000 UTC m=+49.635134185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert") pod "ingress-canary-8pg7p" (UID: "99ba1953-8c33-4ce7-81dd-ff524369264f") : secret "canary-serving-cert" not found
May 28 11:03:53.238480 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.238443    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb"]
May 28 11:03:53.277321 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.277284    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb"]
May 28 11:03:53.277471 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.277333    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb"
May 28 11:03:53.279397 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.279378    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 28 11:03:53.279510 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.279410    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 28 11:03:53.280061 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.280044    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4m64r\""
May 28 11:03:53.280061 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.280048    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 28 11:03:53.280252 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.280140    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 28 11:03:53.449480 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.449444    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0fc1992-1aac-4451-bb27-d21dab64f436-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-656c7bdd74-bk9xb\" (UID: \"d0fc1992-1aac-4451-bb27-d21dab64f436\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb"
May 28 11:03:53.449657 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.449504    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsxmv\" (UniqueName: \"kubernetes.io/projected/d0fc1992-1aac-4451-bb27-d21dab64f436-kube-api-access-wsxmv\") pod \"managed-serviceaccount-addon-agent-656c7bdd74-bk9xb\" (UID: \"d0fc1992-1aac-4451-bb27-d21dab64f436\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb"
May 28 11:03:53.550820 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.550733    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0fc1992-1aac-4451-bb27-d21dab64f436-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-656c7bdd74-bk9xb\" (UID: \"d0fc1992-1aac-4451-bb27-d21dab64f436\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb"
May 28 11:03:53.550820 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.550785    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wsxmv\" (UniqueName: \"kubernetes.io/projected/d0fc1992-1aac-4451-bb27-d21dab64f436-kube-api-access-wsxmv\") pod \"managed-serviceaccount-addon-agent-656c7bdd74-bk9xb\" (UID: \"d0fc1992-1aac-4451-bb27-d21dab64f436\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb"
May 28 11:03:53.554087 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.554062    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0fc1992-1aac-4451-bb27-d21dab64f436-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-656c7bdd74-bk9xb\" (UID: \"d0fc1992-1aac-4451-bb27-d21dab64f436\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb"
May 28 11:03:53.558582 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.558550    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsxmv\" (UniqueName: \"kubernetes.io/projected/d0fc1992-1aac-4451-bb27-d21dab64f436-kube-api-access-wsxmv\") pod \"managed-serviceaccount-addon-agent-656c7bdd74-bk9xb\" (UID: \"d0fc1992-1aac-4451-bb27-d21dab64f436\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb"
May 28 11:03:53.600509 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.600479    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb"
May 28 11:03:53.651684 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.651650    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert\") pod \"ingress-canary-8pg7p\" (UID: \"99ba1953-8c33-4ce7-81dd-ff524369264f\") " pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:03:53.651835 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.651703    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:03:53.651835 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:53.651795    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 11:03:53.651835 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:53.651804    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 11:03:53.651957 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:53.651857    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls podName:566c95b3-6db1-4296-8ec0-f823019d6a40 nodeName:}" failed. No retries permitted until 2026-05-28 11:04:09.651843431 +0000 UTC m=+65.690940387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls") pod "dns-default-7c95j" (UID: "566c95b3-6db1-4296-8ec0-f823019d6a40") : secret "dns-default-metrics-tls" not found
May 28 11:03:53.651957 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:03:53.651889    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert podName:99ba1953-8c33-4ce7-81dd-ff524369264f nodeName:}" failed. No retries permitted until 2026-05-28 11:04:09.651875628 +0000 UTC m=+65.690972585 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert") pod "ingress-canary-8pg7p" (UID: "99ba1953-8c33-4ce7-81dd-ff524369264f") : secret "canary-serving-cert" not found
May 28 11:03:53.774727 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:53.774658    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb"]
May 28 11:03:53.778776 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:03:53.778747    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0fc1992_1aac_4451_bb27_d21dab64f436.slice/crio-fc3a8b32bbb213769366593254683a48a833e3106c5c6ad646b49804644cad5a WatchSource:0}: Error finding container fc3a8b32bbb213769366593254683a48a833e3106c5c6ad646b49804644cad5a: Status 404 returned error can't find the container with id fc3a8b32bbb213769366593254683a48a833e3106c5c6ad646b49804644cad5a
May 28 11:03:54.708279 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:54.708243    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb" event={"ID":"d0fc1992-1aac-4451-bb27-d21dab64f436","Type":"ContainerStarted","Data":"fc3a8b32bbb213769366593254683a48a833e3106c5c6ad646b49804644cad5a"}
May 28 11:03:56.713820 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:56.713787    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb" event={"ID":"d0fc1992-1aac-4451-bb27-d21dab64f436","Type":"ContainerStarted","Data":"66429dcc70f5d84bbd7158751f82b936f4b49d6ac8d53b9dbc13b3243794d80f"}
May 28 11:03:56.729673 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:03:56.729616    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb" podStartSLOduration=0.878422285 podStartE2EDuration="3.729598342s" podCreationTimestamp="2026-05-28 11:03:53 +0000 UTC" firstStartedPulling="2026-05-28 11:03:53.780491978 +0000 UTC m=+49.819588935" lastFinishedPulling="2026-05-28 11:03:56.631668027 +0000 UTC m=+52.670764992" observedRunningTime="2026-05-28 11:03:56.72932328 +0000 UTC m=+52.768420265" watchObservedRunningTime="2026-05-28 11:03:56.729598342 +0000 UTC m=+52.768695323"
May 28 11:04:03.662277 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:03.662245    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fhbhc"
May 28 11:04:09.671716 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:09.671681    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert\") pod \"ingress-canary-8pg7p\" (UID: \"99ba1953-8c33-4ce7-81dd-ff524369264f\") " pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:04:09.672168 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:09.671730    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:04:09.672168 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:09.671849    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 11:04:09.672168 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:09.671925    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert podName:99ba1953-8c33-4ce7-81dd-ff524369264f nodeName:}" failed. No retries permitted until 2026-05-28 11:04:41.671909151 +0000 UTC m=+97.711006108 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert") pod "ingress-canary-8pg7p" (UID: "99ba1953-8c33-4ce7-81dd-ff524369264f") : secret "canary-serving-cert" not found
May 28 11:04:09.672168 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:09.671854    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 11:04:09.672168 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:09.671996    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls podName:566c95b3-6db1-4296-8ec0-f823019d6a40 nodeName:}" failed. No retries permitted until 2026-05-28 11:04:41.671980553 +0000 UTC m=+97.711077513 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls") pod "dns-default-7c95j" (UID: "566c95b3-6db1-4296-8ec0-f823019d6a40") : secret "dns-default-metrics-tls" not found
May 28 11:04:10.176458 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:10.176421    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs\") pod \"network-metrics-daemon-5jcnw\" (UID: \"1f58f4ea-58e7-4a59-afd8-b41076bfa130\") " pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:04:10.176650 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:10.176495    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2gf74\" (UniqueName: \"kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74\") pod \"network-check-target-kc5jm\" (UID: \"c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2\") " pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:04:10.178757 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:10.178738    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 28 11:04:10.187107 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:10.187088    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 28 11:04:10.187203 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:10.187167    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs podName:1f58f4ea-58e7-4a59-afd8-b41076bfa130 nodeName:}" failed. No retries permitted until 2026-05-28 11:05:14.187150849 +0000 UTC m=+130.226247806 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs") pod "network-metrics-daemon-5jcnw" (UID: "1f58f4ea-58e7-4a59-afd8-b41076bfa130") : secret "metrics-daemon-secret" not found
May 28 11:04:10.190546 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:10.190530    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 28 11:04:10.198487 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:10.198471    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 28 11:04:10.210399 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:10.210371    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gf74\" (UniqueName: \"kubernetes.io/projected/c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2-kube-api-access-2gf74\") pod \"network-check-target-kc5jm\" (UID: \"c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2\") " pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:04:10.433474 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:10.433395    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7rhrk\""
May 28 11:04:10.442028 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:10.442005    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:04:10.557769 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:10.557739    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kc5jm"]
May 28 11:04:10.561460 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:04:10.561427    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3baf54a_14a3_45e2_89a8_ddc61fd0b8f2.slice/crio-967dce31d1c77e8a167461bc1495d21db3d2950b82f3c164c1b9545d81bda0be WatchSource:0}: Error finding container 967dce31d1c77e8a167461bc1495d21db3d2950b82f3c164c1b9545d81bda0be: Status 404 returned error can't find the container with id 967dce31d1c77e8a167461bc1495d21db3d2950b82f3c164c1b9545d81bda0be
May 28 11:04:10.741846 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:10.741811    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kc5jm" event={"ID":"c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2","Type":"ContainerStarted","Data":"967dce31d1c77e8a167461bc1495d21db3d2950b82f3c164c1b9545d81bda0be"}
May 28 11:04:13.748750 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:13.748720    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kc5jm" event={"ID":"c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2","Type":"ContainerStarted","Data":"17abd2b31f70001cbbbf9092412d25a87cf49bc4932ee3c9b82ca1e54b927b35"}
May 28 11:04:13.749095 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:13.748861    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:04:13.764005 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:13.763954    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kc5jm" podStartSLOduration=67.116106386 podStartE2EDuration="1m9.763931452s" podCreationTimestamp="2026-05-28 11:03:04 +0000 UTC" firstStartedPulling="2026-05-28 11:04:10.563242856 +0000 UTC m=+66.602339817" lastFinishedPulling="2026-05-28 11:04:13.211067913 +0000 UTC m=+69.250164883" observedRunningTime="2026-05-28 11:04:13.763890113 +0000 UTC m=+69.802987092" watchObservedRunningTime="2026-05-28 11:04:13.763931452 +0000 UTC m=+69.803028431"
May 28 11:04:33.345513 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.345372    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-5lgwk"]
May 28 11:04:33.350445 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.350420    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-c79db4c74-j496b"]
May 28 11:04:33.350581 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.350568    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-5lgwk"
May 28 11:04:33.352792 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.352758    2552 status_manager.go:895] "Failed to get status for pod" podUID="3aa30366-ea47-4dad-ab45-38932e616d71" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-5lgwk" err="pods \"volume-data-source-validator-5cf8c89d64-5lgwk\" is forbidden: User \"system:node:ip-10-0-133-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-cluster-storage-operator\": no relationship found between node 'ip-10-0-133-107.ec2.internal' and this object"
May 28 11:04:33.352903 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.352804    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:33.353333 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:33.353306    2552 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"volume-data-source-validator-dockercfg-ldf46\" is forbidden: User \"system:node:ip-10-0-133-107.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-cluster-storage-operator\": no relationship found between node 'ip-10-0-133-107.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ldf46\"" type="*v1.Secret"
May 28 11:04:33.353432 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:33.353414    2552 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-133-107.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-cluster-storage-operator\": no relationship found between node 'ip-10-0-133-107.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
May 28 11:04:33.353496 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:33.353454    2552 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-133-107.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-cluster-storage-operator\": no relationship found between node 'ip-10-0-133-107.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
May 28 11:04:33.355870 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.355842    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 28 11:04:33.355969 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.355913    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 28 11:04:33.356157 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.356140    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 28 11:04:33.356238 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.356198    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kdvwt\""
May 28 11:04:33.356413 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.356397    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 28 11:04:33.356752 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.356731    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 28 11:04:33.361778 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.361756    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 28 11:04:33.373811 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.373787    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-5lgwk"]
May 28 11:04:33.375693 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.375668    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c79db4c74-j496b"]
May 28 11:04:33.435256 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.435220    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nhmv\" (UniqueName: \"kubernetes.io/projected/3aa30366-ea47-4dad-ab45-38932e616d71-kube-api-access-4nhmv\") pod \"volume-data-source-validator-5cf8c89d64-5lgwk\" (UID: \"3aa30366-ea47-4dad-ab45-38932e616d71\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-5lgwk"
May 28 11:04:33.435256 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.435261    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-stats-auth\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:33.435394 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.435296    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:33.435394 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.435357    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:33.435461 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.435396    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-default-certificate\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:33.435461 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.435419    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgcvq\" (UniqueName: \"kubernetes.io/projected/037f095d-ebe7-44bd-8c85-0902026307ae-kube-api-access-zgcvq\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:33.447084 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.447062    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-s5n2k"]
May 28 11:04:33.448845 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.448831    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68f66dcf74-p9x9l"]
May 28 11:04:33.448977 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.448961    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:33.450349 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.450335    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.452840 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.452825    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt"]
May 28 11:04:33.453412 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.453391    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 28 11:04:33.453504 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.453454    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 28 11:04:33.453575 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.453499    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 28 11:04:33.453575 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.453557    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 28 11:04:33.453733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.453719    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tstrs\""
May 28 11:04:33.454240 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.454221    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 28 11:04:33.454369 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.454253    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qmz7c\""
May 28 11:04:33.454468 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.454409    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"]
May 28 11:04:33.454528 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.454512    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt"
May 28 11:04:33.455543 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.455525    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 28 11:04:33.456411 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.456390    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"
May 28 11:04:33.456411 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.456405    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 28 11:04:33.456562 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.456438    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 28 11:04:33.456562 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.456444    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 28 11:04:33.456562 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.456452    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 28 11:04:33.456772 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.456753    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6gxbh\""
May 28 11:04:33.456900 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.456884    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 28 11:04:33.461608 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.461588    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 28 11:04:33.461725 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.461709    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 28 11:04:33.461963 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.461937    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 28 11:04:33.462109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.462094    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rk9vs\""
May 28 11:04:33.463224 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.463207    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 28 11:04:33.464789 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.464707    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 28 11:04:33.468051 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.468026    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"]
May 28 11:04:33.469103 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.469085    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-s5n2k"]
May 28 11:04:33.477586 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.477566    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68f66dcf74-p9x9l"]
May 28 11:04:33.489294 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.489274    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt"]
May 28 11:04:33.536338 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536311    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99a5f73b-ab15-4705-aa51-6f19d1e0f558-config\") pod \"console-operator-76dff48d59-s5n2k\" (UID: \"99a5f73b-ab15-4705-aa51-6f19d1e0f558\") " pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:33.536463 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536351    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:33.536463 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536376    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99a5f73b-ab15-4705-aa51-6f19d1e0f558-trusted-ca\") pod \"console-operator-76dff48d59-s5n2k\" (UID: \"99a5f73b-ab15-4705-aa51-6f19d1e0f558\") " pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:33.536463 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536405    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-default-certificate\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:33.536463 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536431    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zgcvq\" (UniqueName: \"kubernetes.io/projected/037f095d-ebe7-44bd-8c85-0902026307ae-kube-api-access-zgcvq\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:33.536463 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536456    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b537d95c-acec-49cb-8bd6-ea72063471e2-trusted-ca\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.536706 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:33.536489    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle podName:037f095d-ebe7-44bd-8c85-0902026307ae nodeName:}" failed. No retries permitted until 2026-05-28 11:04:34.036470699 +0000 UTC m=+90.075567658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle") pod "router-default-c79db4c74-j496b" (UID: "037f095d-ebe7-44bd-8c85-0902026307ae") : configmap references non-existent config key: service-ca.crt
May 28 11:04:33.536706 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536515    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-stats-auth\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:33.536706 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536540    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c4865c2f-d302-411e-ac37-42d66f527743-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-vvkxt\" (UID: \"c4865c2f-d302-411e-ac37-42d66f527743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt"
May 28 11:04:33.536706 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536566    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-bound-sa-token\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.536706 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536662    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b537d95c-acec-49cb-8bd6-ea72063471e2-installation-pull-secrets\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.536706 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536699    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzs7v\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-kube-api-access-vzs7v\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.537008 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536716    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9vq7\" (UniqueName: \"kubernetes.io/projected/6bf8f86f-b15e-457d-a25a-9472411ea86d-kube-api-access-k9vq7\") pod \"cluster-samples-operator-697587599-9fhq4\" (UID: \"6bf8f86f-b15e-457d-a25a-9472411ea86d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"
May 28 11:04:33.537008 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536757    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x5nn\" (UniqueName: \"kubernetes.io/projected/c4865c2f-d302-411e-ac37-42d66f527743-kube-api-access-2x5nn\") pod \"kube-storage-version-migrator-operator-565644f49-vvkxt\" (UID: \"c4865c2f-d302-411e-ac37-42d66f527743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt"
May 28 11:04:33.537008 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536793    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:33.537008 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536829    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c4865c2f-d302-411e-ac37-42d66f527743-config\") pod \"kube-storage-version-migrator-operator-565644f49-vvkxt\" (UID: \"c4865c2f-d302-411e-ac37-42d66f527743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt"
May 28 11:04:33.537008 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536874    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v6zj\" (UniqueName: \"kubernetes.io/projected/99a5f73b-ab15-4705-aa51-6f19d1e0f558-kube-api-access-9v6zj\") pod \"console-operator-76dff48d59-s5n2k\" (UID: \"99a5f73b-ab15-4705-aa51-6f19d1e0f558\") " pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:33.537008 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:33.536924    2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 11:04:33.537008 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536930    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/99a5f73b-ab15-4705-aa51-6f19d1e0f558-serving-cert\") pod \"console-operator-76dff48d59-s5n2k\" (UID: \"99a5f73b-ab15-4705-aa51-6f19d1e0f558\") " pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:33.537008 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.536981    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b537d95c-acec-49cb-8bd6-ea72063471e2-image-registry-private-configuration\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.537008 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:33.536992    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs podName:037f095d-ebe7-44bd-8c85-0902026307ae nodeName:}" failed. No retries permitted until 2026-05-28 11:04:34.036974194 +0000 UTC m=+90.076071156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs") pod "router-default-c79db4c74-j496b" (UID: "037f095d-ebe7-44bd-8c85-0902026307ae") : secret "router-metrics-certs-default" not found
May 28 11:04:33.537395 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.537034    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4nhmv\" (UniqueName: \"kubernetes.io/projected/3aa30366-ea47-4dad-ab45-38932e616d71-kube-api-access-4nhmv\") pod \"volume-data-source-validator-5cf8c89d64-5lgwk\" (UID: \"3aa30366-ea47-4dad-ab45-38932e616d71\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-5lgwk"
May 28 11:04:33.537395 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.537068    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9fhq4\" (UID: \"6bf8f86f-b15e-457d-a25a-9472411ea86d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"
May 28 11:04:33.537395 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.537100    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-certificates\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.537395 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.537160    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b537d95c-acec-49cb-8bd6-ea72063471e2-ca-trust-extracted\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.537395 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.537191    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.538866 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.538843    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-stats-auth\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:33.538931 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.538894    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-default-certificate\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:33.556850 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.556830    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgcvq\" (UniqueName: \"kubernetes.io/projected/037f095d-ebe7-44bd-8c85-0902026307ae-kube-api-access-zgcvq\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:33.637828 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.637765    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/99a5f73b-ab15-4705-aa51-6f19d1e0f558-serving-cert\") pod \"console-operator-76dff48d59-s5n2k\" (UID: \"99a5f73b-ab15-4705-aa51-6f19d1e0f558\") " pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:33.637828 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.637799    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b537d95c-acec-49cb-8bd6-ea72063471e2-image-registry-private-configuration\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.637988 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.637831    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9fhq4\" (UID: \"6bf8f86f-b15e-457d-a25a-9472411ea86d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"
May 28 11:04:33.637988 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.637857    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-certificates\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.637988 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.637887    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b537d95c-acec-49cb-8bd6-ea72063471e2-ca-trust-extracted\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.637988 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.637912    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.637988 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.637937    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99a5f73b-ab15-4705-aa51-6f19d1e0f558-config\") pod \"console-operator-76dff48d59-s5n2k\" (UID: \"99a5f73b-ab15-4705-aa51-6f19d1e0f558\") " pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:33.637988 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:33.637961    2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 11:04:33.637988 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.637982    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99a5f73b-ab15-4705-aa51-6f19d1e0f558-trusted-ca\") pod \"console-operator-76dff48d59-s5n2k\" (UID: \"99a5f73b-ab15-4705-aa51-6f19d1e0f558\") " pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:33.638371 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:33.638023    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls podName:6bf8f86f-b15e-457d-a25a-9472411ea86d nodeName:}" failed. No retries permitted until 2026-05-28 11:04:34.138004272 +0000 UTC m=+90.177101247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls") pod "cluster-samples-operator-697587599-9fhq4" (UID: "6bf8f86f-b15e-457d-a25a-9472411ea86d") : secret "samples-operator-tls" not found
May 28 11:04:33.638371 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.638046    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b537d95c-acec-49cb-8bd6-ea72063471e2-trusted-ca\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.638371 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.638085    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c4865c2f-d302-411e-ac37-42d66f527743-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-vvkxt\" (UID: \"c4865c2f-d302-411e-ac37-42d66f527743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt"
May 28 11:04:33.638371 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:33.638090    2552 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 11:04:33.638371 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:33.638109    2552 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f66dcf74-p9x9l: secret "image-registry-tls" not found
May 28 11:04:33.638371 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:33.638176    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls podName:b537d95c-acec-49cb-8bd6-ea72063471e2 nodeName:}" failed. No retries permitted until 2026-05-28 11:04:34.138159788 +0000 UTC m=+90.177256754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls") pod "image-registry-68f66dcf74-p9x9l" (UID: "b537d95c-acec-49cb-8bd6-ea72063471e2") : secret "image-registry-tls" not found
May 28 11:04:33.638681 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.638393    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-bound-sa-token\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.638681 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.638448    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b537d95c-acec-49cb-8bd6-ea72063471e2-installation-pull-secrets\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.638681 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.638494    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vzs7v\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-kube-api-access-vzs7v\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.638681 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.638520    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k9vq7\" (UniqueName: \"kubernetes.io/projected/6bf8f86f-b15e-457d-a25a-9472411ea86d-kube-api-access-k9vq7\") pod \"cluster-samples-operator-697587599-9fhq4\" (UID: \"6bf8f86f-b15e-457d-a25a-9472411ea86d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"
May 28 11:04:33.638681 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.638547    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2x5nn\" (UniqueName: \"kubernetes.io/projected/c4865c2f-d302-411e-ac37-42d66f527743-kube-api-access-2x5nn\") pod \"kube-storage-version-migrator-operator-565644f49-vvkxt\" (UID: \"c4865c2f-d302-411e-ac37-42d66f527743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt"
May 28 11:04:33.638681 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.638594    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c4865c2f-d302-411e-ac37-42d66f527743-config\") pod \"kube-storage-version-migrator-operator-565644f49-vvkxt\" (UID: \"c4865c2f-d302-411e-ac37-42d66f527743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt"
May 28 11:04:33.638681 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.638590    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-certificates\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.638681 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.638638    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9v6zj\" (UniqueName: \"kubernetes.io/projected/99a5f73b-ab15-4705-aa51-6f19d1e0f558-kube-api-access-9v6zj\") pod \"console-operator-76dff48d59-s5n2k\" (UID: \"99a5f73b-ab15-4705-aa51-6f19d1e0f558\") " pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:33.639059 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.638786    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/99a5f73b-ab15-4705-aa51-6f19d1e0f558-config\") pod \"console-operator-76dff48d59-s5n2k\" (UID: \"99a5f73b-ab15-4705-aa51-6f19d1e0f558\") " pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:33.639414 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.639388    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b537d95c-acec-49cb-8bd6-ea72063471e2-trusted-ca\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.639584 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.639561    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b537d95c-acec-49cb-8bd6-ea72063471e2-ca-trust-extracted\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.639875 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.639851    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99a5f73b-ab15-4705-aa51-6f19d1e0f558-trusted-ca\") pod \"console-operator-76dff48d59-s5n2k\" (UID: \"99a5f73b-ab15-4705-aa51-6f19d1e0f558\") " pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:33.640067 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.640044    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c4865c2f-d302-411e-ac37-42d66f527743-config\") pod \"kube-storage-version-migrator-operator-565644f49-vvkxt\" (UID: \"c4865c2f-d302-411e-ac37-42d66f527743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt"
May 28 11:04:33.640689 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.640670    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/99a5f73b-ab15-4705-aa51-6f19d1e0f558-serving-cert\") pod \"console-operator-76dff48d59-s5n2k\" (UID: \"99a5f73b-ab15-4705-aa51-6f19d1e0f558\") " pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:33.640774 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.640703    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b537d95c-acec-49cb-8bd6-ea72063471e2-image-registry-private-configuration\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.640774 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.640746    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c4865c2f-d302-411e-ac37-42d66f527743-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-vvkxt\" (UID: \"c4865c2f-d302-411e-ac37-42d66f527743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt"
May 28 11:04:33.641078 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.641063    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b537d95c-acec-49cb-8bd6-ea72063471e2-installation-pull-secrets\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.646856 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.646837    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v6zj\" (UniqueName: \"kubernetes.io/projected/99a5f73b-ab15-4705-aa51-6f19d1e0f558-kube-api-access-9v6zj\") pod \"console-operator-76dff48d59-s5n2k\" (UID: \"99a5f73b-ab15-4705-aa51-6f19d1e0f558\") " pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:33.654879 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.654848    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzs7v\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-kube-api-access-vzs7v\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.654972 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.654882    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x5nn\" (UniqueName: \"kubernetes.io/projected/c4865c2f-d302-411e-ac37-42d66f527743-kube-api-access-2x5nn\") pod \"kube-storage-version-migrator-operator-565644f49-vvkxt\" (UID: \"c4865c2f-d302-411e-ac37-42d66f527743\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt"
May 28 11:04:33.655354 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.655335    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9vq7\" (UniqueName: \"kubernetes.io/projected/6bf8f86f-b15e-457d-a25a-9472411ea86d-kube-api-access-k9vq7\") pod \"cluster-samples-operator-697587599-9fhq4\" (UID: \"6bf8f86f-b15e-457d-a25a-9472411ea86d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"
May 28 11:04:33.656918 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.656898    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-bound-sa-token\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:33.760143 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.760093    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:33.774050 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.774023    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt"
May 28 11:04:33.884644 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.884584    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-s5n2k"]
May 28 11:04:33.888218 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:04:33.888157    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a5f73b_ab15_4705_aa51_6f19d1e0f558.slice/crio-e9430004c1745946b5f7933e714f29c47ad528004028696fb48b82e95242e332 WatchSource:0}: Error finding container e9430004c1745946b5f7933e714f29c47ad528004028696fb48b82e95242e332: Status 404 returned error can't find the container with id e9430004c1745946b5f7933e714f29c47ad528004028696fb48b82e95242e332
May 28 11:04:33.899561 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:33.899533    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt"]
May 28 11:04:33.903244 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:04:33.903218    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4865c2f_d302_411e_ac37_42d66f527743.slice/crio-b9685172d1c15cc20069a08b2914d9e5256f601c6311872d5f3794f7616dc34d WatchSource:0}: Error finding container b9685172d1c15cc20069a08b2914d9e5256f601c6311872d5f3794f7616dc34d: Status 404 returned error can't find the container with id b9685172d1c15cc20069a08b2914d9e5256f601c6311872d5f3794f7616dc34d
May 28 11:04:34.041795 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:34.041772    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:34.041888 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:34.041851    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:34.041935 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:34.041907    2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 11:04:34.042033 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:34.041965    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs podName:037f095d-ebe7-44bd-8c85-0902026307ae nodeName:}" failed. No retries permitted until 2026-05-28 11:04:35.041946767 +0000 UTC m=+91.081043732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs") pod "router-default-c79db4c74-j496b" (UID: "037f095d-ebe7-44bd-8c85-0902026307ae") : secret "router-metrics-certs-default" not found
May 28 11:04:34.042033 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:34.041980    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle podName:037f095d-ebe7-44bd-8c85-0902026307ae nodeName:}" failed. No retries permitted until 2026-05-28 11:04:35.041973576 +0000 UTC m=+91.081070533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle") pod "router-default-c79db4c74-j496b" (UID: "037f095d-ebe7-44bd-8c85-0902026307ae") : configmap references non-existent config key: service-ca.crt
May 28 11:04:34.142448 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:34.142391    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9fhq4\" (UID: \"6bf8f86f-b15e-457d-a25a-9472411ea86d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"
May 28 11:04:34.142448 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:34.142424    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:34.142556 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:34.142533    2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 11:04:34.142598 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:34.142587    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls podName:6bf8f86f-b15e-457d-a25a-9472411ea86d nodeName:}" failed. No retries permitted until 2026-05-28 11:04:35.142570783 +0000 UTC m=+91.181667757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls") pod "cluster-samples-operator-697587599-9fhq4" (UID: "6bf8f86f-b15e-457d-a25a-9472411ea86d") : secret "samples-operator-tls" not found
May 28 11:04:34.142640 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:34.142607    2552 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 11:04:34.142640 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:34.142622    2552 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f66dcf74-p9x9l: secret "image-registry-tls" not found
May 28 11:04:34.142699 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:34.142656    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls podName:b537d95c-acec-49cb-8bd6-ea72063471e2 nodeName:}" failed. No retries permitted until 2026-05-28 11:04:35.142644826 +0000 UTC m=+91.181741787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls") pod "image-registry-68f66dcf74-p9x9l" (UID: "b537d95c-acec-49cb-8bd6-ea72063471e2") : secret "image-registry-tls" not found
May 28 11:04:34.366354 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:34.366330    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 28 11:04:34.456184 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:34.456095    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ldf46\""
May 28 11:04:34.555396 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:34.555310    2552 projected.go:289] Couldn't get configMap openshift-cluster-storage-operator/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
May 28 11:04:34.555396 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:34.555347    2552 projected.go:194] Error preparing data for projected volume kube-api-access-4nhmv for pod openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-5lgwk: failed to sync configmap cache: timed out waiting for the condition
May 28 11:04:34.555680 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:34.555414    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3aa30366-ea47-4dad-ab45-38932e616d71-kube-api-access-4nhmv podName:3aa30366-ea47-4dad-ab45-38932e616d71 nodeName:}" failed. No retries permitted until 2026-05-28 11:04:35.055395289 +0000 UTC m=+91.094492258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4nhmv" (UniqueName: "kubernetes.io/projected/3aa30366-ea47-4dad-ab45-38932e616d71-kube-api-access-4nhmv") pod "volume-data-source-validator-5cf8c89d64-5lgwk" (UID: "3aa30366-ea47-4dad-ab45-38932e616d71") : failed to sync configmap cache: timed out waiting for the condition
May 28 11:04:34.689801 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:34.689773    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 28 11:04:34.790197 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:34.790160    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k" event={"ID":"99a5f73b-ab15-4705-aa51-6f19d1e0f558","Type":"ContainerStarted","Data":"e9430004c1745946b5f7933e714f29c47ad528004028696fb48b82e95242e332"}
May 28 11:04:34.791464 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:34.791428    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt" event={"ID":"c4865c2f-d302-411e-ac37-42d66f527743","Type":"ContainerStarted","Data":"b9685172d1c15cc20069a08b2914d9e5256f601c6311872d5f3794f7616dc34d"}
May 28 11:04:35.048908 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:35.048824    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:35.048908 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:35.048894    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:35.049092 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:35.049010    2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 11:04:35.049092 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:35.049087    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs podName:037f095d-ebe7-44bd-8c85-0902026307ae nodeName:}" failed. No retries permitted until 2026-05-28 11:04:37.049063071 +0000 UTC m=+93.088160029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs") pod "router-default-c79db4c74-j496b" (UID: "037f095d-ebe7-44bd-8c85-0902026307ae") : secret "router-metrics-certs-default" not found
May 28 11:04:35.049225 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:35.049109    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle podName:037f095d-ebe7-44bd-8c85-0902026307ae nodeName:}" failed. No retries permitted until 2026-05-28 11:04:37.049099734 +0000 UTC m=+93.088196694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle") pod "router-default-c79db4c74-j496b" (UID: "037f095d-ebe7-44bd-8c85-0902026307ae") : configmap references non-existent config key: service-ca.crt
May 28 11:04:35.150300 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:35.150264    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9fhq4\" (UID: \"6bf8f86f-b15e-457d-a25a-9472411ea86d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"
May 28 11:04:35.150478 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:35.150313    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:35.150478 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:35.150436    2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 11:04:35.150478 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:35.150471    2552 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 11:04:35.150606 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:35.150489    2552 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f66dcf74-p9x9l: secret "image-registry-tls" not found
May 28 11:04:35.150606 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:35.150447    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4nhmv\" (UniqueName: \"kubernetes.io/projected/3aa30366-ea47-4dad-ab45-38932e616d71-kube-api-access-4nhmv\") pod \"volume-data-source-validator-5cf8c89d64-5lgwk\" (UID: \"3aa30366-ea47-4dad-ab45-38932e616d71\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-5lgwk"
May 28 11:04:35.150606 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:35.150513    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls podName:6bf8f86f-b15e-457d-a25a-9472411ea86d nodeName:}" failed. No retries permitted until 2026-05-28 11:04:37.150493576 +0000 UTC m=+93.189590534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls") pod "cluster-samples-operator-697587599-9fhq4" (UID: "6bf8f86f-b15e-457d-a25a-9472411ea86d") : secret "samples-operator-tls" not found
May 28 11:04:35.150606 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:35.150572    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls podName:b537d95c-acec-49cb-8bd6-ea72063471e2 nodeName:}" failed. No retries permitted until 2026-05-28 11:04:37.150557812 +0000 UTC m=+93.189654772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls") pod "image-registry-68f66dcf74-p9x9l" (UID: "b537d95c-acec-49cb-8bd6-ea72063471e2") : secret "image-registry-tls" not found
May 28 11:04:35.153044 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:35.153022    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nhmv\" (UniqueName: \"kubernetes.io/projected/3aa30366-ea47-4dad-ab45-38932e616d71-kube-api-access-4nhmv\") pod \"volume-data-source-validator-5cf8c89d64-5lgwk\" (UID: \"3aa30366-ea47-4dad-ab45-38932e616d71\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-5lgwk"
May 28 11:04:35.160074 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:35.160054    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-5lgwk"
May 28 11:04:36.125416 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:36.125353    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-5lgwk"]
May 28 11:04:36.129170 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:04:36.129124    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa30366_ea47_4dad_ab45_38932e616d71.slice/crio-d6484d0c6362775356e422e4379757360ef8f956a74236b5d125ea4d4e12ee96 WatchSource:0}: Error finding container d6484d0c6362775356e422e4379757360ef8f956a74236b5d125ea4d4e12ee96: Status 404 returned error can't find the container with id d6484d0c6362775356e422e4379757360ef8f956a74236b5d125ea4d4e12ee96
May 28 11:04:36.797041 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:36.796998    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt" event={"ID":"c4865c2f-d302-411e-ac37-42d66f527743","Type":"ContainerStarted","Data":"39ed6a9a9840c28134f03c49338e5ee9f6c6dc8b4df890195a54e840bcefda37"}
May 28 11:04:36.798408 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:36.798369    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-5lgwk" event={"ID":"3aa30366-ea47-4dad-ab45-38932e616d71","Type":"ContainerStarted","Data":"d6484d0c6362775356e422e4379757360ef8f956a74236b5d125ea4d4e12ee96"}
May 28 11:04:36.801500 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:36.801477    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/0.log"
May 28 11:04:36.801624 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:36.801519    2552 generic.go:358] "Generic (PLEG): container finished" podID="99a5f73b-ab15-4705-aa51-6f19d1e0f558" containerID="56c601baec15633e382112221afc17e625f6ef2d594981cd89e2f39c54d3f3d6" exitCode=255
May 28 11:04:36.801624 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:36.801552    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k" event={"ID":"99a5f73b-ab15-4705-aa51-6f19d1e0f558","Type":"ContainerDied","Data":"56c601baec15633e382112221afc17e625f6ef2d594981cd89e2f39c54d3f3d6"}
May 28 11:04:36.801855 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:36.801841    2552 scope.go:117] "RemoveContainer" containerID="56c601baec15633e382112221afc17e625f6ef2d594981cd89e2f39c54d3f3d6"
May 28 11:04:36.812975 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:36.812921    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt" podStartSLOduration=1.711042065 podStartE2EDuration="3.812902553s" podCreationTimestamp="2026-05-28 11:04:33 +0000 UTC" firstStartedPulling="2026-05-28 11:04:33.904928749 +0000 UTC m=+89.944025706" lastFinishedPulling="2026-05-28 11:04:36.006789224 +0000 UTC m=+92.045886194" observedRunningTime="2026-05-28 11:04:36.810886425 +0000 UTC m=+92.849983405" watchObservedRunningTime="2026-05-28 11:04:36.812902553 +0000 UTC m=+92.851999536"
May 28 11:04:37.065822 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:37.065735    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:37.065970 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:37.065863    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:37.065970 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:37.065898    2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 11:04:37.066071 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:37.065974    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs podName:037f095d-ebe7-44bd-8c85-0902026307ae nodeName:}" failed. No retries permitted until 2026-05-28 11:04:41.06595258 +0000 UTC m=+97.105049540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs") pod "router-default-c79db4c74-j496b" (UID: "037f095d-ebe7-44bd-8c85-0902026307ae") : secret "router-metrics-certs-default" not found
May 28 11:04:37.066071 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:37.066004    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle podName:037f095d-ebe7-44bd-8c85-0902026307ae nodeName:}" failed. No retries permitted until 2026-05-28 11:04:41.065988416 +0000 UTC m=+97.105085393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle") pod "router-default-c79db4c74-j496b" (UID: "037f095d-ebe7-44bd-8c85-0902026307ae") : configmap references non-existent config key: service-ca.crt
May 28 11:04:37.166313 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:37.166274    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9fhq4\" (UID: \"6bf8f86f-b15e-457d-a25a-9472411ea86d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"
May 28 11:04:37.166703 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:37.166325    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:37.166703 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:37.166414    2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 11:04:37.166703 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:37.166425    2552 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 11:04:37.166703 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:37.166472    2552 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f66dcf74-p9x9l: secret "image-registry-tls" not found
May 28 11:04:37.166703 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:37.166473    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls podName:6bf8f86f-b15e-457d-a25a-9472411ea86d nodeName:}" failed. No retries permitted until 2026-05-28 11:04:41.166459165 +0000 UTC m=+97.205556127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls") pod "cluster-samples-operator-697587599-9fhq4" (UID: "6bf8f86f-b15e-457d-a25a-9472411ea86d") : secret "samples-operator-tls" not found
May 28 11:04:37.166703 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:37.166519    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls podName:b537d95c-acec-49cb-8bd6-ea72063471e2 nodeName:}" failed. No retries permitted until 2026-05-28 11:04:41.16650813 +0000 UTC m=+97.205605090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls") pod "image-registry-68f66dcf74-p9x9l" (UID: "b537d95c-acec-49cb-8bd6-ea72063471e2") : secret "image-registry-tls" not found
May 28 11:04:37.805650 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:37.805574    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/1.log"
May 28 11:04:37.805922 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:37.805907    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/0.log"
May 28 11:04:37.805968 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:37.805944    2552 generic.go:358] "Generic (PLEG): container finished" podID="99a5f73b-ab15-4705-aa51-6f19d1e0f558" containerID="96ec9333b16237e8a1a3df74561edd61dddf2401388b6855567e6f489b423271" exitCode=255
May 28 11:04:37.806062 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:37.806033    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k" event={"ID":"99a5f73b-ab15-4705-aa51-6f19d1e0f558","Type":"ContainerDied","Data":"96ec9333b16237e8a1a3df74561edd61dddf2401388b6855567e6f489b423271"}
May 28 11:04:37.806206 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:37.806080    2552 scope.go:117] "RemoveContainer" containerID="56c601baec15633e382112221afc17e625f6ef2d594981cd89e2f39c54d3f3d6"
May 28 11:04:37.806331 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:37.806312    2552 scope.go:117] "RemoveContainer" containerID="96ec9333b16237e8a1a3df74561edd61dddf2401388b6855567e6f489b423271"
May 28 11:04:37.806544 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:37.806526    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-s5n2k_openshift-console-operator(99a5f73b-ab15-4705-aa51-6f19d1e0f558)\"" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k" podUID="99a5f73b-ab15-4705-aa51-6f19d1e0f558"
May 28 11:04:37.807415 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:37.807266    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-5lgwk" event={"ID":"3aa30366-ea47-4dad-ab45-38932e616d71","Type":"ContainerStarted","Data":"961e800e23fef3f358109448302ef0717b29dc8ead190d9eaaaee1a8025e5805"}
May 28 11:04:37.838273 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:37.838225    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-5lgwk" podStartSLOduration=3.4724962059999998 podStartE2EDuration="4.83821179s" podCreationTimestamp="2026-05-28 11:04:33 +0000 UTC" firstStartedPulling="2026-05-28 11:04:36.131796346 +0000 UTC m=+92.170893318" lastFinishedPulling="2026-05-28 11:04:37.497511941 +0000 UTC m=+93.536608902" observedRunningTime="2026-05-28 11:04:37.837538373 +0000 UTC m=+93.876635353" watchObservedRunningTime="2026-05-28 11:04:37.83821179 +0000 UTC m=+93.877308769"
May 28 11:04:38.810488 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:38.810455    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/1.log"
May 28 11:04:38.814073 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:38.811036    2552 scope.go:117] "RemoveContainer" containerID="96ec9333b16237e8a1a3df74561edd61dddf2401388b6855567e6f489b423271"
May 28 11:04:38.814073 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:38.811399    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-s5n2k_openshift-console-operator(99a5f73b-ab15-4705-aa51-6f19d1e0f558)\"" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k" podUID="99a5f73b-ab15-4705-aa51-6f19d1e0f558"
May 28 11:04:39.001694 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:39.001666    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4lh8f_cba5e87a-42ea-447e-8c21-439ce0bbb9fd/dns-node-resolver/0.log"
May 28 11:04:40.001038 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.001009    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hsvzs_4fd526dd-8e8c-4600-a0cb-182ce3516c6f/node-ca/0.log"
May 28 11:04:40.157595 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.157558    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-hdhh6"]
May 28 11:04:40.159926 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.159910    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-hdhh6"
May 28 11:04:40.161927 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.161903    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 28 11:04:40.162050 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.161928    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 28 11:04:40.162504 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.162479    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 28 11:04:40.162504 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.162478    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pbzn6\""
May 28 11:04:40.162689 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.162523    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 28 11:04:40.167009 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.166990    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-hdhh6"]
May 28 11:04:40.292042 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.291948    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c9492ee6-80d4-41e0-8fab-22a8d6039a3a-signing-cabundle\") pod \"service-ca-67d667c7d-hdhh6\" (UID: \"c9492ee6-80d4-41e0-8fab-22a8d6039a3a\") " pod="openshift-service-ca/service-ca-67d667c7d-hdhh6"
May 28 11:04:40.292042 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.292026    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c9492ee6-80d4-41e0-8fab-22a8d6039a3a-signing-key\") pod \"service-ca-67d667c7d-hdhh6\" (UID: \"c9492ee6-80d4-41e0-8fab-22a8d6039a3a\") " pod="openshift-service-ca/service-ca-67d667c7d-hdhh6"
May 28 11:04:40.292252 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.292054    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bk2x\" (UniqueName: \"kubernetes.io/projected/c9492ee6-80d4-41e0-8fab-22a8d6039a3a-kube-api-access-2bk2x\") pod \"service-ca-67d667c7d-hdhh6\" (UID: \"c9492ee6-80d4-41e0-8fab-22a8d6039a3a\") " pod="openshift-service-ca/service-ca-67d667c7d-hdhh6"
May 28 11:04:40.392837 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.392798    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c9492ee6-80d4-41e0-8fab-22a8d6039a3a-signing-cabundle\") pod \"service-ca-67d667c7d-hdhh6\" (UID: \"c9492ee6-80d4-41e0-8fab-22a8d6039a3a\") " pod="openshift-service-ca/service-ca-67d667c7d-hdhh6"
May 28 11:04:40.393027 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.392870    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c9492ee6-80d4-41e0-8fab-22a8d6039a3a-signing-key\") pod \"service-ca-67d667c7d-hdhh6\" (UID: \"c9492ee6-80d4-41e0-8fab-22a8d6039a3a\") " pod="openshift-service-ca/service-ca-67d667c7d-hdhh6"
May 28 11:04:40.393027 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.392895    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2bk2x\" (UniqueName: \"kubernetes.io/projected/c9492ee6-80d4-41e0-8fab-22a8d6039a3a-kube-api-access-2bk2x\") pod \"service-ca-67d667c7d-hdhh6\" (UID: \"c9492ee6-80d4-41e0-8fab-22a8d6039a3a\") " pod="openshift-service-ca/service-ca-67d667c7d-hdhh6"
May 28 11:04:40.393508 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.393485    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c9492ee6-80d4-41e0-8fab-22a8d6039a3a-signing-cabundle\") pod \"service-ca-67d667c7d-hdhh6\" (UID: \"c9492ee6-80d4-41e0-8fab-22a8d6039a3a\") " pod="openshift-service-ca/service-ca-67d667c7d-hdhh6"
May 28 11:04:40.395221 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.395193    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c9492ee6-80d4-41e0-8fab-22a8d6039a3a-signing-key\") pod \"service-ca-67d667c7d-hdhh6\" (UID: \"c9492ee6-80d4-41e0-8fab-22a8d6039a3a\") " pod="openshift-service-ca/service-ca-67d667c7d-hdhh6"
May 28 11:04:40.401121 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.401096    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bk2x\" (UniqueName: \"kubernetes.io/projected/c9492ee6-80d4-41e0-8fab-22a8d6039a3a-kube-api-access-2bk2x\") pod \"service-ca-67d667c7d-hdhh6\" (UID: \"c9492ee6-80d4-41e0-8fab-22a8d6039a3a\") " pod="openshift-service-ca/service-ca-67d667c7d-hdhh6"
May 28 11:04:40.469195 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.469165    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-hdhh6"
May 28 11:04:40.595504 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.595417    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-hdhh6"]
May 28 11:04:40.600877 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:04:40.600849    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9492ee6_80d4_41e0_8fab_22a8d6039a3a.slice/crio-05517d1e4e0806c190840e009c145f02d961bb6549ff85783722610960bf6fc2 WatchSource:0}: Error finding container 05517d1e4e0806c190840e009c145f02d961bb6549ff85783722610960bf6fc2: Status 404 returned error can't find the container with id 05517d1e4e0806c190840e009c145f02d961bb6549ff85783722610960bf6fc2
May 28 11:04:40.814766 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:40.814734    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-hdhh6" event={"ID":"c9492ee6-80d4-41e0-8fab-22a8d6039a3a","Type":"ContainerStarted","Data":"05517d1e4e0806c190840e009c145f02d961bb6549ff85783722610960bf6fc2"}
May 28 11:04:41.099785 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:41.099753    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:41.100233 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:41.099865    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:41.100233 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:41.099907    2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 11:04:41.100233 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:41.099992    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs podName:037f095d-ebe7-44bd-8c85-0902026307ae nodeName:}" failed. No retries permitted until 2026-05-28 11:04:49.099966323 +0000 UTC m=+105.139063295 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs") pod "router-default-c79db4c74-j496b" (UID: "037f095d-ebe7-44bd-8c85-0902026307ae") : secret "router-metrics-certs-default" not found
May 28 11:04:41.100233 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:41.100008    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle podName:037f095d-ebe7-44bd-8c85-0902026307ae nodeName:}" failed. No retries permitted until 2026-05-28 11:04:49.100002157 +0000 UTC m=+105.139099115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle") pod "router-default-c79db4c74-j496b" (UID: "037f095d-ebe7-44bd-8c85-0902026307ae") : configmap references non-existent config key: service-ca.crt
May 28 11:04:41.200681 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:41.200634    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9fhq4\" (UID: \"6bf8f86f-b15e-457d-a25a-9472411ea86d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"
May 28 11:04:41.200864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:41.200714    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:41.200864 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:41.200764    2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 11:04:41.200864 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:41.200832    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls podName:6bf8f86f-b15e-457d-a25a-9472411ea86d nodeName:}" failed. No retries permitted until 2026-05-28 11:04:49.200813546 +0000 UTC m=+105.239910502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls") pod "cluster-samples-operator-697587599-9fhq4" (UID: "6bf8f86f-b15e-457d-a25a-9472411ea86d") : secret "samples-operator-tls" not found
May 28 11:04:41.201019 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:41.200766    2552 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 28 11:04:41.201019 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:41.200900    2552 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f66dcf74-p9x9l: secret "image-registry-tls" not found
May 28 11:04:41.201019 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:41.200963    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls podName:b537d95c-acec-49cb-8bd6-ea72063471e2 nodeName:}" failed. No retries permitted until 2026-05-28 11:04:49.200946838 +0000 UTC m=+105.240043823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls") pod "image-registry-68f66dcf74-p9x9l" (UID: "b537d95c-acec-49cb-8bd6-ea72063471e2") : secret "image-registry-tls" not found
May 28 11:04:41.705349 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:41.705313    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert\") pod \"ingress-canary-8pg7p\" (UID: \"99ba1953-8c33-4ce7-81dd-ff524369264f\") " pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:04:41.705523 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:41.705437    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:04:41.705523 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:41.705479    2552 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 11:04:41.705618 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:41.705532    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 11:04:41.705618 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:41.705558    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert podName:99ba1953-8c33-4ce7-81dd-ff524369264f nodeName:}" failed. No retries permitted until 2026-05-28 11:05:45.705536653 +0000 UTC m=+161.744633613 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert") pod "ingress-canary-8pg7p" (UID: "99ba1953-8c33-4ce7-81dd-ff524369264f") : secret "canary-serving-cert" not found
May 28 11:04:41.705618 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:41.705580    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls podName:566c95b3-6db1-4296-8ec0-f823019d6a40 nodeName:}" failed. No retries permitted until 2026-05-28 11:05:45.705565128 +0000 UTC m=+161.744662106 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls") pod "dns-default-7c95j" (UID: "566c95b3-6db1-4296-8ec0-f823019d6a40") : secret "dns-default-metrics-tls" not found
May 28 11:04:42.821102 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:42.821067    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-hdhh6" event={"ID":"c9492ee6-80d4-41e0-8fab-22a8d6039a3a","Type":"ContainerStarted","Data":"d1012432c461a7bb914c0e5f9354d933bde6c9f9658cab95b6a89be962027d34"}
May 28 11:04:42.835813 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:42.835761    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-hdhh6" podStartSLOduration=1.220179381 podStartE2EDuration="2.835744338s" podCreationTimestamp="2026-05-28 11:04:40 +0000 UTC" firstStartedPulling="2026-05-28 11:04:40.602611307 +0000 UTC m=+96.641708265" lastFinishedPulling="2026-05-28 11:04:42.218176251 +0000 UTC m=+98.257273222" observedRunningTime="2026-05-28 11:04:42.835622533 +0000 UTC m=+98.874719523" watchObservedRunningTime="2026-05-28 11:04:42.835744338 +0000 UTC m=+98.874841320"
May 28 11:04:43.761210 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:43.761172    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:43.761210 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:43.761212    2552 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:04:43.761574 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:43.761559    2552 scope.go:117] "RemoveContainer" containerID="96ec9333b16237e8a1a3df74561edd61dddf2401388b6855567e6f489b423271"
May 28 11:04:43.761735 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:43.761719    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-s5n2k_openshift-console-operator(99a5f73b-ab15-4705-aa51-6f19d1e0f558)\"" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k" podUID="99a5f73b-ab15-4705-aa51-6f19d1e0f558"
May 28 11:04:44.752760 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:44.752727    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kc5jm"
May 28 11:04:49.169930 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.169894    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:49.170411 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.169960    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:49.170411 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:49.170071    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle podName:037f095d-ebe7-44bd-8c85-0902026307ae nodeName:}" failed. No retries permitted until 2026-05-28 11:05:05.170052105 +0000 UTC m=+121.209149064 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle") pod "router-default-c79db4c74-j496b" (UID: "037f095d-ebe7-44bd-8c85-0902026307ae") : configmap references non-existent config key: service-ca.crt
May 28 11:04:49.172422 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.172394    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037f095d-ebe7-44bd-8c85-0902026307ae-metrics-certs\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:04:49.270536 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.270498    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9fhq4\" (UID: \"6bf8f86f-b15e-457d-a25a-9472411ea86d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"
May 28 11:04:49.270712 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.270553    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:49.272890 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.272865    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6bf8f86f-b15e-457d-a25a-9472411ea86d-samples-operator-tls\") pod \"cluster-samples-operator-697587599-9fhq4\" (UID: \"6bf8f86f-b15e-457d-a25a-9472411ea86d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"
May 28 11:04:49.273042 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.273020    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls\") pod \"image-registry-68f66dcf74-p9x9l\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") " pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:49.367958 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.367931    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:49.379786 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.379761    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"
May 28 11:04:49.510434 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.510402    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68f66dcf74-p9x9l"]
May 28 11:04:49.514370 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:04:49.514345    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb537d95c_acec_49cb_8bd6_ea72063471e2.slice/crio-e19319fa2fad070dc0a95baa32664c6aaa20badccad05b080a19b66ef475e3cf WatchSource:0}: Error finding container e19319fa2fad070dc0a95baa32664c6aaa20badccad05b080a19b66ef475e3cf: Status 404 returned error can't find the container with id e19319fa2fad070dc0a95baa32664c6aaa20badccad05b080a19b66ef475e3cf
May 28 11:04:49.522007 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.521981    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4"]
May 28 11:04:49.840555 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.840520    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l" event={"ID":"b537d95c-acec-49cb-8bd6-ea72063471e2","Type":"ContainerStarted","Data":"7f60fa613be738a6ea12e2c6237bd590f11966be59841f0d28a851e901f79aa4"}
May 28 11:04:49.840555 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.840562    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l" event={"ID":"b537d95c-acec-49cb-8bd6-ea72063471e2","Type":"ContainerStarted","Data":"e19319fa2fad070dc0a95baa32664c6aaa20badccad05b080a19b66ef475e3cf"}
May 28 11:04:49.840828 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.840641    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:04:49.841599 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.841577    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4" event={"ID":"6bf8f86f-b15e-457d-a25a-9472411ea86d","Type":"ContainerStarted","Data":"642b311b5b038eb2aa54a63150c2a2bca062dad76afcfedbe32e5eeac48e1361"}
May 28 11:04:49.858620 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:49.858559    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l" podStartSLOduration=16.858542535 podStartE2EDuration="16.858542535s" podCreationTimestamp="2026-05-28 11:04:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 11:04:49.857165987 +0000 UTC m=+105.896262967" watchObservedRunningTime="2026-05-28 11:04:49.858542535 +0000 UTC m=+105.897639515"
May 28 11:04:51.852365 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:51.852326    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4" event={"ID":"6bf8f86f-b15e-457d-a25a-9472411ea86d","Type":"ContainerStarted","Data":"aefa73e5e41170b1ab0138053686ff0dcde64fe53b43e5faaeb3a9b5ed26e22b"}
May 28 11:04:51.852365 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:51.852370    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4" event={"ID":"6bf8f86f-b15e-457d-a25a-9472411ea86d","Type":"ContainerStarted","Data":"77a4889f1469b9b5ed43598b6bf583c595024cdbe5cd9a1bc9165350c694bc9d"}
May 28 11:04:51.870062 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:51.870010    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-9fhq4" podStartSLOduration=17.211050965 podStartE2EDuration="18.869993296s" podCreationTimestamp="2026-05-28 11:04:33 +0000 UTC" firstStartedPulling="2026-05-28 11:04:49.560571114 +0000 UTC m=+105.599668071" lastFinishedPulling="2026-05-28 11:04:51.219513442 +0000 UTC m=+107.258610402" observedRunningTime="2026-05-28 11:04:51.869007566 +0000 UTC m=+107.908104563" watchObservedRunningTime="2026-05-28 11:04:51.869993296 +0000 UTC m=+107.909090273"
May 28 11:04:59.512980 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:59.512945    2552 scope.go:117] "RemoveContainer" containerID="96ec9333b16237e8a1a3df74561edd61dddf2401388b6855567e6f489b423271"
May 28 11:04:59.871950 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:59.871883    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:04:59.872274 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:59.872259    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/1.log"
May 28 11:04:59.872321 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:59.872292    2552 generic.go:358] "Generic (PLEG): container finished" podID="99a5f73b-ab15-4705-aa51-6f19d1e0f558" containerID="bd9c1af31e9eda6da15942b38564b2ad22b2a8c86f972f47a964d9050eb3c92b" exitCode=255
May 28 11:04:59.872369 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:59.872353    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k" event={"ID":"99a5f73b-ab15-4705-aa51-6f19d1e0f558","Type":"ContainerDied","Data":"bd9c1af31e9eda6da15942b38564b2ad22b2a8c86f972f47a964d9050eb3c92b"}
May 28 11:04:59.872403 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:59.872384    2552 scope.go:117] "RemoveContainer" containerID="96ec9333b16237e8a1a3df74561edd61dddf2401388b6855567e6f489b423271"
May 28 11:04:59.872704 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:04:59.872688    2552 scope.go:117] "RemoveContainer" containerID="bd9c1af31e9eda6da15942b38564b2ad22b2a8c86f972f47a964d9050eb3c92b"
May 28 11:04:59.872897 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:04:59.872880    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-s5n2k_openshift-console-operator(99a5f73b-ab15-4705-aa51-6f19d1e0f558)\"" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k" podUID="99a5f73b-ab15-4705-aa51-6f19d1e0f558"
May 28 11:05:00.876018 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:00.875985    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:05:00.894755 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:00.894726    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-whkx6"]
May 28 11:05:00.897239 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:00.897219    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:00.897900 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:00.897882    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68f66dcf74-p9x9l"]
May 28 11:05:00.901325 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:00.901304    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fs2pg\""
May 28 11:05:00.901325 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:00.901304    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 28 11:05:00.901502 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:00.901330    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 28 11:05:00.901502 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:00.901314    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 28 11:05:00.901771 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:00.901756    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 28 11:05:00.913499 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:00.913475    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-whkx6"]
May 28 11:05:01.065385 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.065358    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a64635e-7734-4e92-8253-2de6548df3c2-data-volume\") pod \"insights-runtime-extractor-whkx6\" (UID: \"1a64635e-7734-4e92-8253-2de6548df3c2\") " pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.065553 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.065427    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a64635e-7734-4e92-8253-2de6548df3c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whkx6\" (UID: \"1a64635e-7734-4e92-8253-2de6548df3c2\") " pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.065553 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.065459    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a64635e-7734-4e92-8253-2de6548df3c2-crio-socket\") pod \"insights-runtime-extractor-whkx6\" (UID: \"1a64635e-7734-4e92-8253-2de6548df3c2\") " pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.065553 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.065499    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a64635e-7734-4e92-8253-2de6548df3c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whkx6\" (UID: \"1a64635e-7734-4e92-8253-2de6548df3c2\") " pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.065553 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.065516    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp5hq\" (UniqueName: \"kubernetes.io/projected/1a64635e-7734-4e92-8253-2de6548df3c2-kube-api-access-qp5hq\") pod \"insights-runtime-extractor-whkx6\" (UID: \"1a64635e-7734-4e92-8253-2de6548df3c2\") " pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.166726 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.166652    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a64635e-7734-4e92-8253-2de6548df3c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whkx6\" (UID: \"1a64635e-7734-4e92-8253-2de6548df3c2\") " pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.166726 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.166682    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qp5hq\" (UniqueName: \"kubernetes.io/projected/1a64635e-7734-4e92-8253-2de6548df3c2-kube-api-access-qp5hq\") pod \"insights-runtime-extractor-whkx6\" (UID: \"1a64635e-7734-4e92-8253-2de6548df3c2\") " pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.166726 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.166707    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a64635e-7734-4e92-8253-2de6548df3c2-data-volume\") pod \"insights-runtime-extractor-whkx6\" (UID: \"1a64635e-7734-4e92-8253-2de6548df3c2\") " pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.166979 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.166864    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a64635e-7734-4e92-8253-2de6548df3c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whkx6\" (UID: \"1a64635e-7734-4e92-8253-2de6548df3c2\") " pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.166979 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.166897    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a64635e-7734-4e92-8253-2de6548df3c2-crio-socket\") pod \"insights-runtime-extractor-whkx6\" (UID: \"1a64635e-7734-4e92-8253-2de6548df3c2\") " pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.167075 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.166989    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a64635e-7734-4e92-8253-2de6548df3c2-crio-socket\") pod \"insights-runtime-extractor-whkx6\" (UID: \"1a64635e-7734-4e92-8253-2de6548df3c2\") " pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.167075 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.167022    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a64635e-7734-4e92-8253-2de6548df3c2-data-volume\") pod \"insights-runtime-extractor-whkx6\" (UID: \"1a64635e-7734-4e92-8253-2de6548df3c2\") " pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.167336 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.167321    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a64635e-7734-4e92-8253-2de6548df3c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whkx6\" (UID: \"1a64635e-7734-4e92-8253-2de6548df3c2\") " pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.168958 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.168939    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a64635e-7734-4e92-8253-2de6548df3c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whkx6\" (UID: \"1a64635e-7734-4e92-8253-2de6548df3c2\") " pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.174733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.174711    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp5hq\" (UniqueName: \"kubernetes.io/projected/1a64635e-7734-4e92-8253-2de6548df3c2-kube-api-access-qp5hq\") pod \"insights-runtime-extractor-whkx6\" (UID: \"1a64635e-7734-4e92-8253-2de6548df3c2\") " pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.206207 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.206187    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-whkx6"
May 28 11:05:01.324234 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.324207    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-whkx6"]
May 28 11:05:01.328548 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:05:01.328517    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a64635e_7734_4e92_8253_2de6548df3c2.slice/crio-b7cb0a4295babba606416cf184f265b82c30e0550fb0a6714bf2b5f29ce53797 WatchSource:0}: Error finding container b7cb0a4295babba606416cf184f265b82c30e0550fb0a6714bf2b5f29ce53797: Status 404 returned error can't find the container with id b7cb0a4295babba606416cf184f265b82c30e0550fb0a6714bf2b5f29ce53797
May 28 11:05:01.879746 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.879710    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-whkx6" event={"ID":"1a64635e-7734-4e92-8253-2de6548df3c2","Type":"ContainerStarted","Data":"bef8f379dd3cfe5cf3dac3f0f889186d4558135e3c5e268e32df8095f1cbc134"}
May 28 11:05:01.879746 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:01.879750    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-whkx6" event={"ID":"1a64635e-7734-4e92-8253-2de6548df3c2","Type":"ContainerStarted","Data":"b7cb0a4295babba606416cf184f265b82c30e0550fb0a6714bf2b5f29ce53797"}
May 28 11:05:02.884090 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:02.884054    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-whkx6" event={"ID":"1a64635e-7734-4e92-8253-2de6548df3c2","Type":"ContainerStarted","Data":"4634f3fd168e79d1d7c14b679b9fd9f5d3c21c09b78581d192cb4f5075f1a477"}
May 28 11:05:03.760478 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:03.760457    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:05:03.760585 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:03.760485    2552 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:05:03.760822 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:03.760807    2552 scope.go:117] "RemoveContainer" containerID="bd9c1af31e9eda6da15942b38564b2ad22b2a8c86f972f47a964d9050eb3c92b"
May 28 11:05:03.761000 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:05:03.760981    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-s5n2k_openshift-console-operator(99a5f73b-ab15-4705-aa51-6f19d1e0f558)\"" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k" podUID="99a5f73b-ab15-4705-aa51-6f19d1e0f558"
May 28 11:05:03.890641 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:03.890597    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-whkx6" event={"ID":"1a64635e-7734-4e92-8253-2de6548df3c2","Type":"ContainerStarted","Data":"c8419a467624608c7cfc2d6b26c4537c9c8c5af8bc4175e76402e6b59597eb63"}
May 28 11:05:03.911270 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:03.911224    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-whkx6" podStartSLOduration=1.596903131 podStartE2EDuration="3.911206358s" podCreationTimestamp="2026-05-28 11:05:00 +0000 UTC" firstStartedPulling="2026-05-28 11:05:01.37927841 +0000 UTC m=+117.418375367" lastFinishedPulling="2026-05-28 11:05:03.693581634 +0000 UTC m=+119.732678594" observedRunningTime="2026-05-28 11:05:03.910067952 +0000 UTC m=+119.949164942" watchObservedRunningTime="2026-05-28 11:05:03.911206358 +0000 UTC m=+119.950303337"
May 28 11:05:05.198859 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:05.198823    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:05:05.199411 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:05.199391    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/037f095d-ebe7-44bd-8c85-0902026307ae-service-ca-bundle\") pod \"router-default-c79db4c74-j496b\" (UID: \"037f095d-ebe7-44bd-8c85-0902026307ae\") " pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:05:05.467947 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:05.467883    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kdvwt\""
May 28 11:05:05.476804 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:05.476784    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:05:05.604684 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:05.604654    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c79db4c74-j496b"]
May 28 11:05:05.607826 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:05:05.607791    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037f095d_ebe7_44bd_8c85_0902026307ae.slice/crio-4255b667768230f46d067960dc762de733010c164bc7101be2eaf9469c43f378 WatchSource:0}: Error finding container 4255b667768230f46d067960dc762de733010c164bc7101be2eaf9469c43f378: Status 404 returned error can't find the container with id 4255b667768230f46d067960dc762de733010c164bc7101be2eaf9469c43f378
May 28 11:05:05.897384 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:05.897345    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c79db4c74-j496b" event={"ID":"037f095d-ebe7-44bd-8c85-0902026307ae","Type":"ContainerStarted","Data":"ff98965216c47a60652146c5d20908748bb4d7a626426f7621a4ba20ff423411"}
May 28 11:05:05.897384 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:05.897389    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c79db4c74-j496b" event={"ID":"037f095d-ebe7-44bd-8c85-0902026307ae","Type":"ContainerStarted","Data":"4255b667768230f46d067960dc762de733010c164bc7101be2eaf9469c43f378"}
May 28 11:05:05.915834 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:05.915783    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-c79db4c74-j496b" podStartSLOduration=32.915767536 podStartE2EDuration="32.915767536s" podCreationTimestamp="2026-05-28 11:04:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 11:05:05.914652776 +0000 UTC m=+121.953749757" watchObservedRunningTime="2026-05-28 11:05:05.915767536 +0000 UTC m=+121.954864514"
May 28 11:05:06.477035 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:06.476996    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:05:06.479515 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:06.479482    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:05:06.900101 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:06.900025    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:05:06.901202 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:06.901182    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-c79db4c74-j496b"
May 28 11:05:10.903148 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:10.903103    2552 patch_prober.go:28] interesting pod/image-registry-68f66dcf74-p9x9l container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 28 11:05:10.903604 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:10.903184    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l" podUID="b537d95c-acec-49cb-8bd6-ea72063471e2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 28 11:05:11.483267 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.483231    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"]
May 28 11:05:11.486115 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.486091    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"
May 28 11:05:11.488422 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.488397    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 28 11:05:11.488592 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.488570    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 28 11:05:11.489398 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.489379    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 28 11:05:11.489495 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.489381    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 28 11:05:11.489495 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.489386    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jlggk\""
May 28 11:05:11.489495 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.489386    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 28 11:05:11.493873 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.493852    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"]
May 28 11:05:11.643931 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.643897    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s72kw\" (UniqueName: \"kubernetes.io/projected/37406e38-ff29-456b-bc48-581238e8e9e1-kube-api-access-s72kw\") pod \"prometheus-operator-7779bbdd78-mf2pp\" (UID: \"37406e38-ff29-456b-bc48-581238e8e9e1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"
May 28 11:05:11.644112 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.643960    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37406e38-ff29-456b-bc48-581238e8e9e1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-mf2pp\" (UID: \"37406e38-ff29-456b-bc48-581238e8e9e1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"
May 28 11:05:11.644112 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.644007    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37406e38-ff29-456b-bc48-581238e8e9e1-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-mf2pp\" (UID: \"37406e38-ff29-456b-bc48-581238e8e9e1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"
May 28 11:05:11.644112 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.644042    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37406e38-ff29-456b-bc48-581238e8e9e1-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-mf2pp\" (UID: \"37406e38-ff29-456b-bc48-581238e8e9e1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"
May 28 11:05:11.744708 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.744639    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s72kw\" (UniqueName: \"kubernetes.io/projected/37406e38-ff29-456b-bc48-581238e8e9e1-kube-api-access-s72kw\") pod \"prometheus-operator-7779bbdd78-mf2pp\" (UID: \"37406e38-ff29-456b-bc48-581238e8e9e1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"
May 28 11:05:11.744708 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.744691    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37406e38-ff29-456b-bc48-581238e8e9e1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-mf2pp\" (UID: \"37406e38-ff29-456b-bc48-581238e8e9e1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"
May 28 11:05:11.744859 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.744729    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37406e38-ff29-456b-bc48-581238e8e9e1-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-mf2pp\" (UID: \"37406e38-ff29-456b-bc48-581238e8e9e1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"
May 28 11:05:11.744859 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.744761    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37406e38-ff29-456b-bc48-581238e8e9e1-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-mf2pp\" (UID: \"37406e38-ff29-456b-bc48-581238e8e9e1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"
May 28 11:05:11.745476 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.745452    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37406e38-ff29-456b-bc48-581238e8e9e1-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-mf2pp\" (UID: \"37406e38-ff29-456b-bc48-581238e8e9e1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"
May 28 11:05:11.746872 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.746844    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37406e38-ff29-456b-bc48-581238e8e9e1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-mf2pp\" (UID: \"37406e38-ff29-456b-bc48-581238e8e9e1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"
May 28 11:05:11.746988 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.746972    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/37406e38-ff29-456b-bc48-581238e8e9e1-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-mf2pp\" (UID: \"37406e38-ff29-456b-bc48-581238e8e9e1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"
May 28 11:05:11.753376 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.753351    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s72kw\" (UniqueName: \"kubernetes.io/projected/37406e38-ff29-456b-bc48-581238e8e9e1-kube-api-access-s72kw\") pod \"prometheus-operator-7779bbdd78-mf2pp\" (UID: \"37406e38-ff29-456b-bc48-581238e8e9e1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"
May 28 11:05:11.795370 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.795346    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"
May 28 11:05:11.915724 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:11.915702    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp"]
May 28 11:05:11.919655 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:05:11.918206    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37406e38_ff29_456b_bc48_581238e8e9e1.slice/crio-2292cc697edba415f5b20f56a311cb622a101381723df91c2b7bf546e8445307 WatchSource:0}: Error finding container 2292cc697edba415f5b20f56a311cb622a101381723df91c2b7bf546e8445307: Status 404 returned error can't find the container with id 2292cc697edba415f5b20f56a311cb622a101381723df91c2b7bf546e8445307
May 28 11:05:12.915624 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:12.915588    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp" event={"ID":"37406e38-ff29-456b-bc48-581238e8e9e1","Type":"ContainerStarted","Data":"2292cc697edba415f5b20f56a311cb622a101381723df91c2b7bf546e8445307"}
May 28 11:05:13.919862 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:13.919831    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp" event={"ID":"37406e38-ff29-456b-bc48-581238e8e9e1","Type":"ContainerStarted","Data":"913bbcc558049abe0400287aa9a85fff6ff6f052df844847c5df145ff52776fa"}
May 28 11:05:13.919862 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:13.919865    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp" event={"ID":"37406e38-ff29-456b-bc48-581238e8e9e1","Type":"ContainerStarted","Data":"91da6d31b377deb9ab482c328478bb0f6209fcb271c67be8ad76005083bcf698"}
May 28 11:05:13.949980 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:13.949936    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mf2pp" podStartSLOduration=1.5967657339999999 podStartE2EDuration="2.949920351s" podCreationTimestamp="2026-05-28 11:05:11 +0000 UTC" firstStartedPulling="2026-05-28 11:05:11.92177819 +0000 UTC m=+127.960875148" lastFinishedPulling="2026-05-28 11:05:13.274932808 +0000 UTC m=+129.314029765" observedRunningTime="2026-05-28 11:05:13.949271859 +0000 UTC m=+129.988368837" watchObservedRunningTime="2026-05-28 11:05:13.949920351 +0000 UTC m=+129.989017331"
May 28 11:05:14.264145 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:14.264098    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs\") pod \"network-metrics-daemon-5jcnw\" (UID: \"1f58f4ea-58e7-4a59-afd8-b41076bfa130\") " pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:05:14.266362 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:14.266341    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f58f4ea-58e7-4a59-afd8-b41076bfa130-metrics-certs\") pod \"network-metrics-daemon-5jcnw\" (UID: \"1f58f4ea-58e7-4a59-afd8-b41076bfa130\") " pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:05:14.327910 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:14.327879    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nncjx\""
May 28 11:05:14.335808 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:14.335780    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcnw"
May 28 11:05:14.459026 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:14.459003    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5jcnw"]
May 28 11:05:14.461459 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:05:14.461423    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f58f4ea_58e7_4a59_afd8_b41076bfa130.slice/crio-f9bb46997b2bbde3548d4566dd44ecfb331c338ae78331b8bc8b646b889bb75f WatchSource:0}: Error finding container f9bb46997b2bbde3548d4566dd44ecfb331c338ae78331b8bc8b646b889bb75f: Status 404 returned error can't find the container with id f9bb46997b2bbde3548d4566dd44ecfb331c338ae78331b8bc8b646b889bb75f
May 28 11:05:14.515661 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:14.515601    2552 scope.go:117] "RemoveContainer" containerID="bd9c1af31e9eda6da15942b38564b2ad22b2a8c86f972f47a964d9050eb3c92b"
May 28 11:05:14.515789 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:05:14.515772    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-s5n2k_openshift-console-operator(99a5f73b-ab15-4705-aa51-6f19d1e0f558)\"" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k" podUID="99a5f73b-ab15-4705-aa51-6f19d1e0f558"
May 28 11:05:14.923880 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:14.923793    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5jcnw" event={"ID":"1f58f4ea-58e7-4a59-afd8-b41076bfa130","Type":"ContainerStarted","Data":"f9bb46997b2bbde3548d4566dd44ecfb331c338ae78331b8bc8b646b889bb75f"}
May 28 11:05:15.861708 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.861672    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-clc5r"]
May 28 11:05:15.864185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.864166    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.866404 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.866374    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4w7jl\""
May 28 11:05:15.866658 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.866635    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 28 11:05:15.867112 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.867095    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 28 11:05:15.867224 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.867172    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 28 11:05:15.876704 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.876669    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1987da86-b332-4655-b7d5-5c1061fd900a-metrics-client-ca\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.876812 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.876713    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89kd5\" (UniqueName: \"kubernetes.io/projected/1987da86-b332-4655-b7d5-5c1061fd900a-kube-api-access-89kd5\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.876812 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.876759    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1987da86-b332-4655-b7d5-5c1061fd900a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.876812 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.876788    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1987da86-b332-4655-b7d5-5c1061fd900a-node-exporter-accelerators-collector-config\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.877026 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.876832    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1987da86-b332-4655-b7d5-5c1061fd900a-root\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.877026 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.876857    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1987da86-b332-4655-b7d5-5c1061fd900a-node-exporter-tls\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.877026 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.876884    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1987da86-b332-4655-b7d5-5c1061fd900a-sys\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.877026 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.876905    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1987da86-b332-4655-b7d5-5c1061fd900a-node-exporter-textfile\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.877026 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.876931    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1987da86-b332-4655-b7d5-5c1061fd900a-node-exporter-wtmp\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.928259 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.928218    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5jcnw" event={"ID":"1f58f4ea-58e7-4a59-afd8-b41076bfa130","Type":"ContainerStarted","Data":"a51fae27b72d0a92a8f31bf03ed0252d9ad02dabd1f04211af99297ef81b2137"}
May 28 11:05:15.928259 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.928262    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5jcnw" event={"ID":"1f58f4ea-58e7-4a59-afd8-b41076bfa130","Type":"ContainerStarted","Data":"132896c1dec7217bd30e210a03234b33b4355a13a78c6e5445d197080f1c2731"}
May 28 11:05:15.945409 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.945300    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5jcnw" podStartSLOduration=131.111652836 podStartE2EDuration="2m11.945281875s" podCreationTimestamp="2026-05-28 11:03:04 +0000 UTC" firstStartedPulling="2026-05-28 11:05:14.463365229 +0000 UTC m=+130.502462192" lastFinishedPulling="2026-05-28 11:05:15.296994266 +0000 UTC m=+131.336091231" observedRunningTime="2026-05-28 11:05:15.944647646 +0000 UTC m=+131.983744624" watchObservedRunningTime="2026-05-28 11:05:15.945281875 +0000 UTC m=+131.984378855"
May 28 11:05:15.977401 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.977366    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1987da86-b332-4655-b7d5-5c1061fd900a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.977587 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.977415    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1987da86-b332-4655-b7d5-5c1061fd900a-node-exporter-accelerators-collector-config\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.977587 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.977461    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1987da86-b332-4655-b7d5-5c1061fd900a-root\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.977587 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.977489    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1987da86-b332-4655-b7d5-5c1061fd900a-node-exporter-tls\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.977587 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.977517    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1987da86-b332-4655-b7d5-5c1061fd900a-sys\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.977587 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.977540    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1987da86-b332-4655-b7d5-5c1061fd900a-node-exporter-textfile\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.977587 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.977569    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1987da86-b332-4655-b7d5-5c1061fd900a-node-exporter-wtmp\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.977887 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.977586    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1987da86-b332-4655-b7d5-5c1061fd900a-root\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.977887 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.977684    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1987da86-b332-4655-b7d5-5c1061fd900a-metrics-client-ca\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.977887 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.977715    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-89kd5\" (UniqueName: \"kubernetes.io/projected/1987da86-b332-4655-b7d5-5c1061fd900a-kube-api-access-89kd5\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.978300 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.978121    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1987da86-b332-4655-b7d5-5c1061fd900a-sys\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.978300 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.978255    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1987da86-b332-4655-b7d5-5c1061fd900a-node-exporter-wtmp\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.979151 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.978569    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1987da86-b332-4655-b7d5-5c1061fd900a-node-exporter-textfile\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.979151 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.978590    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1987da86-b332-4655-b7d5-5c1061fd900a-metrics-client-ca\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.979151 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.978867    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1987da86-b332-4655-b7d5-5c1061fd900a-node-exporter-accelerators-collector-config\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.980434 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.980336    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1987da86-b332-4655-b7d5-5c1061fd900a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.980784 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.980764    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1987da86-b332-4655-b7d5-5c1061fd900a-node-exporter-tls\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:15.985043 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:15.985022    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89kd5\" (UniqueName: \"kubernetes.io/projected/1987da86-b332-4655-b7d5-5c1061fd900a-kube-api-access-89kd5\") pod \"node-exporter-clc5r\" (UID: \"1987da86-b332-4655-b7d5-5c1061fd900a\") " pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:16.174177 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:16.174078    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-clc5r"
May 28 11:05:16.932709 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:16.932672    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-clc5r" event={"ID":"1987da86-b332-4655-b7d5-5c1061fd900a","Type":"ContainerStarted","Data":"72a10f02404e1564c99d4892b13f6b83c04dde552933dbcc4e6c740fb29a3302"}
May 28 11:05:16.950070 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:16.950038    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 11:05:16.953548 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:16.953527    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:16.956558 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:16.956528    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 28 11:05:16.956657 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:16.956531    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fc62h\""
May 28 11:05:16.956657 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:16.956531    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 28 11:05:16.956780 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:16.956532    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 28 11:05:16.958098 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:16.958072    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 28 11:05:16.958471 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:16.958273    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 28 11:05:16.958471 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:16.958301    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 28 11:05:16.958471 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:16.958255    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 28 11:05:16.958471 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:16.958402    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 28 11:05:16.963304 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:16.963287    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 28 11:05:16.971271 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:16.968640    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 11:05:17.084511 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.084474    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/76c49803-48d5-4664-8cae-caaa0542a6d1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.084675 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.084519    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76c49803-48d5-4664-8cae-caaa0542a6d1-config-out\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.084675 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.084548    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-web-config\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.084675 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.084568    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vglr9\" (UniqueName: \"kubernetes.io/projected/76c49803-48d5-4664-8cae-caaa0542a6d1-kube-api-access-vglr9\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.084675 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.084635    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c49803-48d5-4664-8cae-caaa0542a6d1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.084815 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.084676    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.084815 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.084702    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-config-volume\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.084815 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.084731    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76c49803-48d5-4664-8cae-caaa0542a6d1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.084815 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.084780    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.084815 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.084799    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76c49803-48d5-4664-8cae-caaa0542a6d1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.084972 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.084838    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.084972 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.084854    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.084972 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.084873    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.185629 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.185570    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/76c49803-48d5-4664-8cae-caaa0542a6d1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.185629 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.185606    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76c49803-48d5-4664-8cae-caaa0542a6d1-config-out\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.185629 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.185624    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-web-config\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.185846 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.185644    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vglr9\" (UniqueName: \"kubernetes.io/projected/76c49803-48d5-4664-8cae-caaa0542a6d1-kube-api-access-vglr9\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.185846 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.185677    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c49803-48d5-4664-8cae-caaa0542a6d1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.185846 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.185709    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.185846 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.185739    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-config-volume\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.185846 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.185766    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76c49803-48d5-4664-8cae-caaa0542a6d1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.185846 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.185803    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.185846 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.185828    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76c49803-48d5-4664-8cae-caaa0542a6d1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.186204 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.185874    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.186204 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.185917    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.186204 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.185949    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.186204 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.185994    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/76c49803-48d5-4664-8cae-caaa0542a6d1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.187114 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.186773    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c49803-48d5-4664-8cae-caaa0542a6d1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.187269 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.187246    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76c49803-48d5-4664-8cae-caaa0542a6d1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.188690 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.188666    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.188914 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.188890    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76c49803-48d5-4664-8cae-caaa0542a6d1-config-out\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.189033 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.188994    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.189096 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.189032    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.189232 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.189195    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-web-config\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.189542 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.189524    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.189696 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.189672    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76c49803-48d5-4664-8cae-caaa0542a6d1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.190055 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.190031    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.190274 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.190256    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-config-volume\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.193437 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.193418    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vglr9\" (UniqueName: \"kubernetes.io/projected/76c49803-48d5-4664-8cae-caaa0542a6d1-kube-api-access-vglr9\") pod \"alertmanager-main-0\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.265849 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.265827    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 11:05:17.396184 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.396158    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 11:05:17.398496 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:05:17.398466    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c49803_48d5_4664_8cae_caaa0542a6d1.slice/crio-39fda053d7819140d789268045ea29d0f3d52f63074749d6cd2ea39e73d0100c WatchSource:0}: Error finding container 39fda053d7819140d789268045ea29d0f3d52f63074749d6cd2ea39e73d0100c: Status 404 returned error can't find the container with id 39fda053d7819140d789268045ea29d0f3d52f63074749d6cd2ea39e73d0100c
May 28 11:05:17.937668 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.937629    2552 generic.go:358] "Generic (PLEG): container finished" podID="1987da86-b332-4655-b7d5-5c1061fd900a" containerID="95d21df7d7b0aed8d0bdbec600f7eca751673bea3ea91e3758b7bb9964164a70" exitCode=0
May 28 11:05:17.938154 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.937691    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-clc5r" event={"ID":"1987da86-b332-4655-b7d5-5c1061fd900a","Type":"ContainerDied","Data":"95d21df7d7b0aed8d0bdbec600f7eca751673bea3ea91e3758b7bb9964164a70"}
May 28 11:05:17.939097 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:17.939076    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c49803-48d5-4664-8cae-caaa0542a6d1","Type":"ContainerStarted","Data":"39fda053d7819140d789268045ea29d0f3d52f63074749d6cd2ea39e73d0100c"}
May 28 11:05:18.945185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:18.944411    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-clc5r" event={"ID":"1987da86-b332-4655-b7d5-5c1061fd900a","Type":"ContainerStarted","Data":"fa274da164fbf39b11274a224afb8485b3fd5674baab50e5c6015fad1aa87322"}
May 28 11:05:18.945185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:18.944451    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-clc5r" event={"ID":"1987da86-b332-4655-b7d5-5c1061fd900a","Type":"ContainerStarted","Data":"4968b3a0ea36f6f3c20b1d0aef141f1dc4b312d2e5c3637dfeb4bb94e8915f9c"}
May 28 11:05:18.946391 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:18.946335    2552 generic.go:358] "Generic (PLEG): container finished" podID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerID="a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd" exitCode=0
May 28 11:05:18.946523 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:18.946423    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c49803-48d5-4664-8cae-caaa0542a6d1","Type":"ContainerDied","Data":"a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd"}
May 28 11:05:18.963145 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:18.963086    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-clc5r" podStartSLOduration=3.046835013 podStartE2EDuration="3.963074318s" podCreationTimestamp="2026-05-28 11:05:15 +0000 UTC" firstStartedPulling="2026-05-28 11:05:16.186106313 +0000 UTC m=+132.225203272" lastFinishedPulling="2026-05-28 11:05:17.102345612 +0000 UTC m=+133.141442577" observedRunningTime="2026-05-28 11:05:18.962228995 +0000 UTC m=+135.001325976" watchObservedRunningTime="2026-05-28 11:05:18.963074318 +0000 UTC m=+135.002171297"
May 28 11:05:20.604972 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:20.604944    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-ps5wd"]
May 28 11:05:20.607483 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:20.607416    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-ps5wd"
May 28 11:05:20.611360 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:20.611285    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d04a73f-8af3-447c-b32f-2d22e86dac3a-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-ps5wd\" (UID: \"1d04a73f-8af3-447c-b32f-2d22e86dac3a\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-ps5wd"
May 28 11:05:20.611726 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:20.611577    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 28 11:05:20.611726 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:20.611594    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-64djv\""
May 28 11:05:20.616673 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:20.616633    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-ps5wd"]
May 28 11:05:20.711846 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:20.711823    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d04a73f-8af3-447c-b32f-2d22e86dac3a-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-ps5wd\" (UID: \"1d04a73f-8af3-447c-b32f-2d22e86dac3a\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-ps5wd"
May 28 11:05:20.711957 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:05:20.711943    2552 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
May 28 11:05:20.712015 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:05:20.711997    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d04a73f-8af3-447c-b32f-2d22e86dac3a-monitoring-plugin-cert podName:1d04a73f-8af3-447c-b32f-2d22e86dac3a nodeName:}" failed. No retries permitted until 2026-05-28 11:05:21.211983104 +0000 UTC m=+137.251080066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/1d04a73f-8af3-447c-b32f-2d22e86dac3a-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-ps5wd" (UID: "1d04a73f-8af3-447c-b32f-2d22e86dac3a") : secret "monitoring-plugin-cert" not found
May 28 11:05:20.901900 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:20.901873    2552 patch_prober.go:28] interesting pod/image-registry-68f66dcf74-p9x9l container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 28 11:05:20.902005 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:20.901922    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l" podUID="b537d95c-acec-49cb-8bd6-ea72063471e2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 28 11:05:20.957515 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:20.957488    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c49803-48d5-4664-8cae-caaa0542a6d1","Type":"ContainerStarted","Data":"62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625"}
May 28 11:05:20.957624 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:20.957522    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c49803-48d5-4664-8cae-caaa0542a6d1","Type":"ContainerStarted","Data":"d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3"}
May 28 11:05:20.957624 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:20.957534    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c49803-48d5-4664-8cae-caaa0542a6d1","Type":"ContainerStarted","Data":"c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e"}
May 28 11:05:20.957624 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:20.957543    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c49803-48d5-4664-8cae-caaa0542a6d1","Type":"ContainerStarted","Data":"0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd"}
May 28 11:05:20.957624 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:20.957553    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c49803-48d5-4664-8cae-caaa0542a6d1","Type":"ContainerStarted","Data":"5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf"}
May 28 11:05:21.217726 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:21.217645    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d04a73f-8af3-447c-b32f-2d22e86dac3a-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-ps5wd\" (UID: \"1d04a73f-8af3-447c-b32f-2d22e86dac3a\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-ps5wd"
May 28 11:05:21.220703 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:21.220660    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d04a73f-8af3-447c-b32f-2d22e86dac3a-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-ps5wd\" (UID: \"1d04a73f-8af3-447c-b32f-2d22e86dac3a\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-ps5wd"
May 28 11:05:21.225294 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:21.225266    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-ps5wd"
May 28 11:05:21.357377 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:21.357342    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-ps5wd"]
May 28 11:05:21.389497 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:05:21.389468    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d04a73f_8af3_447c_b32f_2d22e86dac3a.slice/crio-8132a4342435ac055029e8aa4a0f10c890dfdb1432ea0eca939144b5cec8e0be WatchSource:0}: Error finding container 8132a4342435ac055029e8aa4a0f10c890dfdb1432ea0eca939144b5cec8e0be: Status 404 returned error can't find the container with id 8132a4342435ac055029e8aa4a0f10c890dfdb1432ea0eca939144b5cec8e0be
May 28 11:05:21.963242 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:21.963202    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c49803-48d5-4664-8cae-caaa0542a6d1","Type":"ContainerStarted","Data":"0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343"}
May 28 11:05:21.964487 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:21.964451    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-ps5wd" event={"ID":"1d04a73f-8af3-447c-b32f-2d22e86dac3a","Type":"ContainerStarted","Data":"8132a4342435ac055029e8aa4a0f10c890dfdb1432ea0eca939144b5cec8e0be"}
May 28 11:05:21.993568 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:21.993510    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.982190179 podStartE2EDuration="5.993489615s" podCreationTimestamp="2026-05-28 11:05:16 +0000 UTC" firstStartedPulling="2026-05-28 11:05:17.400456609 +0000 UTC m=+133.439553566" lastFinishedPulling="2026-05-28 11:05:21.411756032 +0000 UTC m=+137.450853002" observedRunningTime="2026-05-28 11:05:21.990918841 +0000 UTC m=+138.030015822" watchObservedRunningTime="2026-05-28 11:05:21.993489615 +0000 UTC m=+138.032586596"
May 28 11:05:23.971618 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:23.971583    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-ps5wd" event={"ID":"1d04a73f-8af3-447c-b32f-2d22e86dac3a","Type":"ContainerStarted","Data":"5acebb94e18cb567e3f6915837ba01a80db03b263550a0091668918d62ea601a"}
May 28 11:05:23.972061 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:23.971801    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-ps5wd"
May 28 11:05:23.976679 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:23.976656    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-ps5wd"
May 28 11:05:23.987593 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:23.987555    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-ps5wd" podStartSLOduration=2.377341941 podStartE2EDuration="3.987541625s" podCreationTimestamp="2026-05-28 11:05:20 +0000 UTC" firstStartedPulling="2026-05-28 11:05:21.391327087 +0000 UTC m=+137.430424044" lastFinishedPulling="2026-05-28 11:05:23.001526768 +0000 UTC m=+139.040623728" observedRunningTime="2026-05-28 11:05:23.985741859 +0000 UTC m=+140.024838837" watchObservedRunningTime="2026-05-28 11:05:23.987541625 +0000 UTC m=+140.026638603"
May 28 11:05:25.917944 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:25.917899    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l" podUID="b537d95c-acec-49cb-8bd6-ea72063471e2" containerName="registry" containerID="cri-o://7f60fa613be738a6ea12e2c6237bd590f11966be59841f0d28a851e901f79aa4" gracePeriod=30
May 28 11:05:26.146385 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.146363    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:05:26.261231 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.261201    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b537d95c-acec-49cb-8bd6-ea72063471e2-installation-pull-secrets\") pod \"b537d95c-acec-49cb-8bd6-ea72063471e2\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") "
May 28 11:05:26.261231 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.261232    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-certificates\") pod \"b537d95c-acec-49cb-8bd6-ea72063471e2\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") "
May 28 11:05:26.261450 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.261270    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls\") pod \"b537d95c-acec-49cb-8bd6-ea72063471e2\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") "
May 28 11:05:26.261450 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.261302    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzs7v\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-kube-api-access-vzs7v\") pod \"b537d95c-acec-49cb-8bd6-ea72063471e2\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") "
May 28 11:05:26.261450 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.261328    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b537d95c-acec-49cb-8bd6-ea72063471e2-image-registry-private-configuration\") pod \"b537d95c-acec-49cb-8bd6-ea72063471e2\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") "
May 28 11:05:26.261450 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.261358    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-bound-sa-token\") pod \"b537d95c-acec-49cb-8bd6-ea72063471e2\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") "
May 28 11:05:26.261450 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.261383    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b537d95c-acec-49cb-8bd6-ea72063471e2-ca-trust-extracted\") pod \"b537d95c-acec-49cb-8bd6-ea72063471e2\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") "
May 28 11:05:26.261450 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.261440    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b537d95c-acec-49cb-8bd6-ea72063471e2-trusted-ca\") pod \"b537d95c-acec-49cb-8bd6-ea72063471e2\" (UID: \"b537d95c-acec-49cb-8bd6-ea72063471e2\") "
May 28 11:05:26.261892 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.261695    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b537d95c-acec-49cb-8bd6-ea72063471e2" (UID: "b537d95c-acec-49cb-8bd6-ea72063471e2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 11:05:26.262207 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.262178    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b537d95c-acec-49cb-8bd6-ea72063471e2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b537d95c-acec-49cb-8bd6-ea72063471e2" (UID: "b537d95c-acec-49cb-8bd6-ea72063471e2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 11:05:26.263857 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.263830    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b537d95c-acec-49cb-8bd6-ea72063471e2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b537d95c-acec-49cb-8bd6-ea72063471e2" (UID: "b537d95c-acec-49cb-8bd6-ea72063471e2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:05:26.264109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.264065    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-kube-api-access-vzs7v" (OuterVolumeSpecName: "kube-api-access-vzs7v") pod "b537d95c-acec-49cb-8bd6-ea72063471e2" (UID: "b537d95c-acec-49cb-8bd6-ea72063471e2"). InnerVolumeSpecName "kube-api-access-vzs7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:05:26.264206 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.264177    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b537d95c-acec-49cb-8bd6-ea72063471e2" (UID: "b537d95c-acec-49cb-8bd6-ea72063471e2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:05:26.264206 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.264177    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b537d95c-acec-49cb-8bd6-ea72063471e2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b537d95c-acec-49cb-8bd6-ea72063471e2" (UID: "b537d95c-acec-49cb-8bd6-ea72063471e2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:05:26.264206 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.264171    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b537d95c-acec-49cb-8bd6-ea72063471e2" (UID: "b537d95c-acec-49cb-8bd6-ea72063471e2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:05:26.269751 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.269721    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b537d95c-acec-49cb-8bd6-ea72063471e2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b537d95c-acec-49cb-8bd6-ea72063471e2" (UID: "b537d95c-acec-49cb-8bd6-ea72063471e2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 11:05:26.362852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.362819    2552 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b537d95c-acec-49cb-8bd6-ea72063471e2-installation-pull-secrets\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:05:26.362852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.362849    2552 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-certificates\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:05:26.363027 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.362863    2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-registry-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:05:26.363027 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.362875    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzs7v\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-kube-api-access-vzs7v\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:05:26.363027 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.362891    2552 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b537d95c-acec-49cb-8bd6-ea72063471e2-image-registry-private-configuration\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:05:26.363027 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.362904    2552 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b537d95c-acec-49cb-8bd6-ea72063471e2-bound-sa-token\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:05:26.363027 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.362917    2552 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b537d95c-acec-49cb-8bd6-ea72063471e2-ca-trust-extracted\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:05:26.363027 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.362929    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b537d95c-acec-49cb-8bd6-ea72063471e2-trusted-ca\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:05:26.514041 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.513964    2552 scope.go:117] "RemoveContainer" containerID="bd9c1af31e9eda6da15942b38564b2ad22b2a8c86f972f47a964d9050eb3c92b"
May 28 11:05:26.927162 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.927058    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-llgn8"]
May 28 11:05:26.927493 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.927403    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b537d95c-acec-49cb-8bd6-ea72063471e2" containerName="registry"
May 28 11:05:26.927493 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.927415    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b537d95c-acec-49cb-8bd6-ea72063471e2" containerName="registry"
May 28 11:05:26.927493 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.927464    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="b537d95c-acec-49cb-8bd6-ea72063471e2" containerName="registry"
May 28 11:05:26.929381 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.929358    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-llgn8"
May 28 11:05:26.931546 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.931520    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 28 11:05:26.931642 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.931599    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 28 11:05:26.931712 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.931522    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-jm6dx\""
May 28 11:05:26.939390 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.939358    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-llgn8"]
May 28 11:05:26.968085 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.968044    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khckw\" (UniqueName: \"kubernetes.io/projected/150584db-a4a1-496c-b65a-33ca9280dd10-kube-api-access-khckw\") pod \"downloads-57bd96b888-llgn8\" (UID: \"150584db-a4a1-496c-b65a-33ca9280dd10\") " pod="openshift-console/downloads-57bd96b888-llgn8"
May 28 11:05:26.980760 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.980724    2552 generic.go:358] "Generic (PLEG): container finished" podID="b537d95c-acec-49cb-8bd6-ea72063471e2" containerID="7f60fa613be738a6ea12e2c6237bd590f11966be59841f0d28a851e901f79aa4" exitCode=0
May 28 11:05:26.980911 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.980836    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l"
May 28 11:05:26.980911 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.980841    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l" event={"ID":"b537d95c-acec-49cb-8bd6-ea72063471e2","Type":"ContainerDied","Data":"7f60fa613be738a6ea12e2c6237bd590f11966be59841f0d28a851e901f79aa4"}
May 28 11:05:26.980911 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.980880    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f66dcf74-p9x9l" event={"ID":"b537d95c-acec-49cb-8bd6-ea72063471e2","Type":"ContainerDied","Data":"e19319fa2fad070dc0a95baa32664c6aaa20badccad05b080a19b66ef475e3cf"}
May 28 11:05:26.981058 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.980923    2552 scope.go:117] "RemoveContainer" containerID="7f60fa613be738a6ea12e2c6237bd590f11966be59841f0d28a851e901f79aa4"
May 28 11:05:26.983790 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.983773    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:05:26.983905 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.983883    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k" event={"ID":"99a5f73b-ab15-4705-aa51-6f19d1e0f558","Type":"ContainerStarted","Data":"713da78fa9bf32c542f8f5f4bab51a710dcf3e5e4165282ca9388be8dfa802e8"}
May 28 11:05:26.984362 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.984340    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:05:26.989669 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.989634    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k"
May 28 11:05:26.991158 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.991112    2552 scope.go:117] "RemoveContainer" containerID="7f60fa613be738a6ea12e2c6237bd590f11966be59841f0d28a851e901f79aa4"
May 28 11:05:26.991426 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:05:26.991408    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f60fa613be738a6ea12e2c6237bd590f11966be59841f0d28a851e901f79aa4\": container with ID starting with 7f60fa613be738a6ea12e2c6237bd590f11966be59841f0d28a851e901f79aa4 not found: ID does not exist" containerID="7f60fa613be738a6ea12e2c6237bd590f11966be59841f0d28a851e901f79aa4"
May 28 11:05:26.991490 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:26.991435    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f60fa613be738a6ea12e2c6237bd590f11966be59841f0d28a851e901f79aa4"} err="failed to get container status \"7f60fa613be738a6ea12e2c6237bd590f11966be59841f0d28a851e901f79aa4\": rpc error: code = NotFound desc = could not find container \"7f60fa613be738a6ea12e2c6237bd590f11966be59841f0d28a851e901f79aa4\": container with ID starting with 7f60fa613be738a6ea12e2c6237bd590f11966be59841f0d28a851e901f79aa4 not found: ID does not exist"
May 28 11:05:27.001249 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:27.001199    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-s5n2k" podStartSLOduration=51.879109316 podStartE2EDuration="54.001184368s" podCreationTimestamp="2026-05-28 11:04:33 +0000 UTC" firstStartedPulling="2026-05-28 11:04:33.890013539 +0000 UTC m=+89.929110495" lastFinishedPulling="2026-05-28 11:04:36.012088586 +0000 UTC m=+92.051185547" observedRunningTime="2026-05-28 11:05:27.000320487 +0000 UTC m=+143.039417492" watchObservedRunningTime="2026-05-28 11:05:27.001184368 +0000 UTC m=+143.040281372"
May 28 11:05:27.012356 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:27.012331    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68f66dcf74-p9x9l"]
May 28 11:05:27.017175 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:27.017154    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68f66dcf74-p9x9l"]
May 28 11:05:27.069503 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:27.069395    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-khckw\" (UniqueName: \"kubernetes.io/projected/150584db-a4a1-496c-b65a-33ca9280dd10-kube-api-access-khckw\") pod \"downloads-57bd96b888-llgn8\" (UID: \"150584db-a4a1-496c-b65a-33ca9280dd10\") " pod="openshift-console/downloads-57bd96b888-llgn8"
May 28 11:05:27.080238 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:27.080209    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khckw\" (UniqueName: \"kubernetes.io/projected/150584db-a4a1-496c-b65a-33ca9280dd10-kube-api-access-khckw\") pod \"downloads-57bd96b888-llgn8\" (UID: \"150584db-a4a1-496c-b65a-33ca9280dd10\") " pod="openshift-console/downloads-57bd96b888-llgn8"
May 28 11:05:27.243489 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:27.243464    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-llgn8"
May 28 11:05:27.365666 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:27.365639    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-llgn8"]
May 28 11:05:27.368098 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:05:27.368067    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150584db_a4a1_496c_b65a_33ca9280dd10.slice/crio-02adaa32af7c6ba925a020809cf1ad7a1bbcbf54e25c8ea55680e575ac34fb09 WatchSource:0}: Error finding container 02adaa32af7c6ba925a020809cf1ad7a1bbcbf54e25c8ea55680e575ac34fb09: Status 404 returned error can't find the container with id 02adaa32af7c6ba925a020809cf1ad7a1bbcbf54e25c8ea55680e575ac34fb09
May 28 11:05:27.989601 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:27.989560    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-llgn8" event={"ID":"150584db-a4a1-496c-b65a-33ca9280dd10","Type":"ContainerStarted","Data":"02adaa32af7c6ba925a020809cf1ad7a1bbcbf54e25c8ea55680e575ac34fb09"}
May 28 11:05:28.517804 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:28.517767    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b537d95c-acec-49cb-8bd6-ea72063471e2" path="/var/lib/kubelet/pods/b537d95c-acec-49cb-8bd6-ea72063471e2/volumes"
May 28 11:05:40.862981 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:05:40.862931    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7c95j" podUID="566c95b3-6db1-4296-8ec0-f823019d6a40"
May 28 11:05:40.869216 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:05:40.869173    2552 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-8pg7p" podUID="99ba1953-8c33-4ce7-81dd-ff524369264f"
May 28 11:05:41.026756 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:41.026718    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:05:41.026756 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:41.026737    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7c95j"
May 28 11:05:44.419212 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.419115    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-66d9b758d6-l5r6m"]
May 28 11:05:44.430931 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.430898    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.433048 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.433011    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 28 11:05:44.433208 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.433036    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66d9b758d6-l5r6m"]
May 28 11:05:44.433463 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.433435    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dz6rq\""
May 28 11:05:44.433567 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.433513    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 28 11:05:44.433567 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.433535    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 28 11:05:44.433767 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.433744    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 28 11:05:44.433996 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.433973    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 28 11:05:44.438586 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.438417    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 28 11:05:44.526730 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.526696    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5487a12f-8342-410a-8c8d-523fadb4b31b-console-serving-cert\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.526730 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.526736    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5wwz\" (UniqueName: \"kubernetes.io/projected/5487a12f-8342-410a-8c8d-523fadb4b31b-kube-api-access-d5wwz\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.526965 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.526766    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-service-ca\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.526965 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.526814    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-trusted-ca-bundle\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.526965 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.526844    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5487a12f-8342-410a-8c8d-523fadb4b31b-console-oauth-config\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.526965 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.526942    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-oauth-serving-cert\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.527154 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.527035    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-console-config\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.627543 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.627503    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5487a12f-8342-410a-8c8d-523fadb4b31b-console-serving-cert\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.627728 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.627556    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d5wwz\" (UniqueName: \"kubernetes.io/projected/5487a12f-8342-410a-8c8d-523fadb4b31b-kube-api-access-d5wwz\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.627728 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.627586    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-service-ca\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.627728 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.627614    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-trusted-ca-bundle\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.627728 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.627644    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5487a12f-8342-410a-8c8d-523fadb4b31b-console-oauth-config\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.627728 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.627681    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-oauth-serving-cert\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.627993 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.627732    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-console-config\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.628536 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.628503    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-service-ca\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.628651 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.628515    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-console-config\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.628784 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.628754    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-trusted-ca-bundle\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.628862 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.628754    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-oauth-serving-cert\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.630820 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.630791    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5487a12f-8342-410a-8c8d-523fadb4b31b-console-oauth-config\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.630974 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.630948    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5487a12f-8342-410a-8c8d-523fadb4b31b-console-serving-cert\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.635927 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.635906    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5wwz\" (UniqueName: \"kubernetes.io/projected/5487a12f-8342-410a-8c8d-523fadb4b31b-kube-api-access-d5wwz\") pod \"console-66d9b758d6-l5r6m\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") " pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.744106 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.744061    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:44.934510 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:44.933723    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66d9b758d6-l5r6m"]
May 28 11:05:44.936519 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:05:44.936490    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5487a12f_8342_410a_8c8d_523fadb4b31b.slice/crio-3bdb05abbc0fe4523602b8b7ab90cd7618c875a848cb8e76082094783ba8dc77 WatchSource:0}: Error finding container 3bdb05abbc0fe4523602b8b7ab90cd7618c875a848cb8e76082094783ba8dc77: Status 404 returned error can't find the container with id 3bdb05abbc0fe4523602b8b7ab90cd7618c875a848cb8e76082094783ba8dc77
May 28 11:05:45.041118 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:45.041007    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-llgn8" event={"ID":"150584db-a4a1-496c-b65a-33ca9280dd10","Type":"ContainerStarted","Data":"6031fb6a660c5e5d2bcea56550c3bdc961a67125294eadfaed5ab0a55af23470"}
May 28 11:05:45.041447 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:45.041397    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-llgn8"
May 28 11:05:45.042367 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:45.042328    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d9b758d6-l5r6m" event={"ID":"5487a12f-8342-410a-8c8d-523fadb4b31b","Type":"ContainerStarted","Data":"3bdb05abbc0fe4523602b8b7ab90cd7618c875a848cb8e76082094783ba8dc77"}
May 28 11:05:45.057261 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:45.057208    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-llgn8" podStartSLOduration=2.336368296 podStartE2EDuration="19.057189692s" podCreationTimestamp="2026-05-28 11:05:26 +0000 UTC" firstStartedPulling="2026-05-28 11:05:27.370165663 +0000 UTC m=+143.409262623" lastFinishedPulling="2026-05-28 11:05:44.090987062 +0000 UTC m=+160.130084019" observedRunningTime="2026-05-28 11:05:45.055575081 +0000 UTC m=+161.094672060" watchObservedRunningTime="2026-05-28 11:05:45.057189692 +0000 UTC m=+161.096286671"
May 28 11:05:45.059066 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:45.059042    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-llgn8"
May 28 11:05:45.739261 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:45.739204    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:05:45.739840 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:45.739288    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert\") pod \"ingress-canary-8pg7p\" (UID: \"99ba1953-8c33-4ce7-81dd-ff524369264f\") " pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:05:45.742454 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:45.742398    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566c95b3-6db1-4296-8ec0-f823019d6a40-metrics-tls\") pod \"dns-default-7c95j\" (UID: \"566c95b3-6db1-4296-8ec0-f823019d6a40\") " pod="openshift-dns/dns-default-7c95j"
May 28 11:05:45.742588 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:45.742472    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ba1953-8c33-4ce7-81dd-ff524369264f-cert\") pod \"ingress-canary-8pg7p\" (UID: \"99ba1953-8c33-4ce7-81dd-ff524369264f\") " pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:05:45.829316 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:45.829282    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nq6bj\""
May 28 11:05:45.829652 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:45.829622    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8gjbt\""
May 28 11:05:45.838584 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:45.838205    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7c95j"
May 28 11:05:45.838584 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:45.838220    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8pg7p"
May 28 11:05:46.039342 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:46.039220    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7c95j"]
May 28 11:05:46.044180 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:05:46.044080    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566c95b3_6db1_4296_8ec0_f823019d6a40.slice/crio-85f5705a29503840e49fe8e0dd3d7817e9892492a67b6c45ebb7f04cda749cf0 WatchSource:0}: Error finding container 85f5705a29503840e49fe8e0dd3d7817e9892492a67b6c45ebb7f04cda749cf0: Status 404 returned error can't find the container with id 85f5705a29503840e49fe8e0dd3d7817e9892492a67b6c45ebb7f04cda749cf0
May 28 11:05:46.050293 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:46.050268    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8pg7p"]
May 28 11:05:46.054447 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:05:46.054414    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ba1953_8c33_4ce7_81dd_ff524369264f.slice/crio-4b9bcee8cd859ceb841edb6568057a077092ce3e032e966eb2a09219c678127e WatchSource:0}: Error finding container 4b9bcee8cd859ceb841edb6568057a077092ce3e032e966eb2a09219c678127e: Status 404 returned error can't find the container with id 4b9bcee8cd859ceb841edb6568057a077092ce3e032e966eb2a09219c678127e
May 28 11:05:47.054910 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:47.054754    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7c95j" event={"ID":"566c95b3-6db1-4296-8ec0-f823019d6a40","Type":"ContainerStarted","Data":"85f5705a29503840e49fe8e0dd3d7817e9892492a67b6c45ebb7f04cda749cf0"}
May 28 11:05:47.057856 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:47.057740    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8pg7p" event={"ID":"99ba1953-8c33-4ce7-81dd-ff524369264f","Type":"ContainerStarted","Data":"4b9bcee8cd859ceb841edb6568057a077092ce3e032e966eb2a09219c678127e"}
May 28 11:05:49.650428 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:49.650332    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76c49803-48d5-4664-8cae-caaa0542a6d1/init-config-reloader/0.log"
May 28 11:05:49.846207 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:49.846122    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76c49803-48d5-4664-8cae-caaa0542a6d1/alertmanager/0.log"
May 28 11:05:50.047191 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:50.047146    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76c49803-48d5-4664-8cae-caaa0542a6d1/config-reloader/0.log"
May 28 11:05:50.245438 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:50.245404    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76c49803-48d5-4664-8cae-caaa0542a6d1/kube-rbac-proxy-web/0.log"
May 28 11:05:50.447426 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:50.447319    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76c49803-48d5-4664-8cae-caaa0542a6d1/kube-rbac-proxy/0.log"
May 28 11:05:50.644765 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:50.644732    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76c49803-48d5-4664-8cae-caaa0542a6d1/kube-rbac-proxy-metric/0.log"
May 28 11:05:50.847475 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:50.847447    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76c49803-48d5-4664-8cae-caaa0542a6d1/prom-label-proxy/0.log"
May 28 11:05:51.071493 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:51.071456    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d9b758d6-l5r6m" event={"ID":"5487a12f-8342-410a-8c8d-523fadb4b31b","Type":"ContainerStarted","Data":"b483950f076472eb61cf104d4e7509bd44182126c987ef2368f15d96ad7b7912"}
May 28 11:05:51.073052 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:51.073021    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8pg7p" event={"ID":"99ba1953-8c33-4ce7-81dd-ff524369264f","Type":"ContainerStarted","Data":"74c5ab66cd722e9e1e661cf18be632521be0d942e701f2e7a2cdd0953d1b9b0f"}
May 28 11:05:51.075188 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:51.075156    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7c95j" event={"ID":"566c95b3-6db1-4296-8ec0-f823019d6a40","Type":"ContainerStarted","Data":"2079d20e9de736c3025a2924641a368189126b90f7379373db817748b8ad3253"}
May 28 11:05:51.075321 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:51.075193    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7c95j" event={"ID":"566c95b3-6db1-4296-8ec0-f823019d6a40","Type":"ContainerStarted","Data":"4d8a0ad93ab1b58c0020838d85a8ae6cecbd07b5abe86c11c7a8616c6fcd1013"}
May 28 11:05:51.075321 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:51.075284    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7c95j"
May 28 11:05:51.090457 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:51.090388    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-66d9b758d6-l5r6m" podStartSLOduration=1.678978587 podStartE2EDuration="7.090376172s" podCreationTimestamp="2026-05-28 11:05:44 +0000 UTC" firstStartedPulling="2026-05-28 11:05:44.939036699 +0000 UTC m=+160.978133655" lastFinishedPulling="2026-05-28 11:05:50.350434284 +0000 UTC m=+166.389531240" observedRunningTime="2026-05-28 11:05:51.088206723 +0000 UTC m=+167.127303704" watchObservedRunningTime="2026-05-28 11:05:51.090376172 +0000 UTC m=+167.129473152"
May 28 11:05:51.103189 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:51.103145    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-7c95j" podStartSLOduration=129.799327936 podStartE2EDuration="2m14.103117811s" podCreationTimestamp="2026-05-28 11:03:37 +0000 UTC" firstStartedPulling="2026-05-28 11:05:46.046600465 +0000 UTC m=+162.085697424" lastFinishedPulling="2026-05-28 11:05:50.350390322 +0000 UTC m=+166.389487299" observedRunningTime="2026-05-28 11:05:51.102383618 +0000 UTC m=+167.141480597" watchObservedRunningTime="2026-05-28 11:05:51.103117811 +0000 UTC m=+167.142214790"
May 28 11:05:51.115965 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:51.115913    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8pg7p" podStartSLOduration=129.817306221 podStartE2EDuration="2m14.115898559s" podCreationTimestamp="2026-05-28 11:03:37 +0000 UTC" firstStartedPulling="2026-05-28 11:05:46.057524216 +0000 UTC m=+162.096621189" lastFinishedPulling="2026-05-28 11:05:50.356116567 +0000 UTC m=+166.395213527" observedRunningTime="2026-05-28 11:05:51.115365942 +0000 UTC m=+167.154462934" watchObservedRunningTime="2026-05-28 11:05:51.115898559 +0000 UTC m=+167.154995539"
May 28 11:05:52.044619 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:52.044579    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-ps5wd_1d04a73f-8af3-447c-b32f-2d22e86dac3a/monitoring-plugin/0.log"
May 28 11:05:52.079973 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:52.079945    2552 generic.go:358] "Generic (PLEG): container finished" podID="c4865c2f-d302-411e-ac37-42d66f527743" containerID="39ed6a9a9840c28134f03c49338e5ee9f6c6dc8b4df890195a54e840bcefda37" exitCode=0
May 28 11:05:52.080147 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:52.080044    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt" event={"ID":"c4865c2f-d302-411e-ac37-42d66f527743","Type":"ContainerDied","Data":"39ed6a9a9840c28134f03c49338e5ee9f6c6dc8b4df890195a54e840bcefda37"}
May 28 11:05:52.080514 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:52.080488    2552 scope.go:117] "RemoveContainer" containerID="39ed6a9a9840c28134f03c49338e5ee9f6c6dc8b4df890195a54e840bcefda37"
May 28 11:05:53.084281 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:53.084245    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-vvkxt" event={"ID":"c4865c2f-d302-411e-ac37-42d66f527743","Type":"ContainerStarted","Data":"7735471f19dcfb6f8d91b493608c6f2e510edc92acd7810412bc0bb0868fd1a8"}
May 28 11:05:53.445223 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:53.445147    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-clc5r_1987da86-b332-4655-b7d5-5c1061fd900a/init-textfile/0.log"
May 28 11:05:53.647464 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:53.647435    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-clc5r_1987da86-b332-4655-b7d5-5c1061fd900a/node-exporter/0.log"
May 28 11:05:53.844899 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:53.844876    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-clc5r_1987da86-b332-4655-b7d5-5c1061fd900a/kube-rbac-proxy/0.log"
May 28 11:05:54.744334 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:54.744296    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:54.744793 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:54.744348    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:05:54.745650 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:54.745627    2552 patch_prober.go:28] interesting pod/console-66d9b758d6-l5r6m container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.20:8443/health\": dial tcp 10.134.0.20:8443: connect: connection refused" start-of-body=
May 28 11:05:54.745741 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:54.745673    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-66d9b758d6-l5r6m" podUID="5487a12f-8342-410a-8c8d-523fadb4b31b" containerName="console" probeResult="failure" output="Get \"https://10.134.0.20:8443/health\": dial tcp 10.134.0.20:8443: connect: connection refused"
May 28 11:05:56.048976 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:56.048946    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-mf2pp_37406e38-ff29-456b-bc48-581238e8e9e1/prometheus-operator/0.log"
May 28 11:05:56.245405 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:56.245369    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-mf2pp_37406e38-ff29-456b-bc48-581238e8e9e1/kube-rbac-proxy/0.log"
May 28 11:05:57.096368 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:57.096291    2552 generic.go:358] "Generic (PLEG): container finished" podID="d0fc1992-1aac-4451-bb27-d21dab64f436" containerID="66429dcc70f5d84bbd7158751f82b936f4b49d6ac8d53b9dbc13b3243794d80f" exitCode=255
May 28 11:05:57.096789 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:57.096363    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb" event={"ID":"d0fc1992-1aac-4451-bb27-d21dab64f436","Type":"ContainerDied","Data":"66429dcc70f5d84bbd7158751f82b936f4b49d6ac8d53b9dbc13b3243794d80f"}
May 28 11:05:57.102370 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:57.102341    2552 scope.go:117] "RemoveContainer" containerID="66429dcc70f5d84bbd7158751f82b936f4b49d6ac8d53b9dbc13b3243794d80f"
May 28 11:05:58.044058 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:58.044031    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:05:58.101252 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:58.101222    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656c7bdd74-bk9xb" event={"ID":"d0fc1992-1aac-4451-bb27-d21dab64f436","Type":"ContainerStarted","Data":"843b4baa191621016edb2ca41dc4321708c95888f8e424ef739522d71e25317b"}
May 28 11:05:58.245392 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:58.245367    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/3.log"
May 28 11:05:58.447114 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:58.447051    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-66d9b758d6-l5r6m_5487a12f-8342-410a-8c8d-523fadb4b31b/console/0.log"
May 28 11:05:58.848169 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:58.848125    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-llgn8_150584db-a4a1-496c-b65a-33ca9280dd10/download-server/0.log"
May 28 11:05:59.445071 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:59.445010    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7c95j_566c95b3-6db1-4296-8ec0-f823019d6a40/dns/0.log"
May 28 11:05:59.646122 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:05:59.646098    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7c95j_566c95b3-6db1-4296-8ec0-f823019d6a40/kube-rbac-proxy/0.log"
May 28 11:06:00.244025 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:00.243999    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4lh8f_cba5e87a-42ea-447e-8c21-439ce0bbb9fd/dns-node-resolver/0.log"
May 28 11:06:01.082244 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:01.082217    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7c95j"
May 28 11:06:01.245964 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:01.245937    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hsvzs_4fd526dd-8e8c-4600-a0cb-182ce3516c6f/node-ca/0.log"
May 28 11:06:01.845121 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:01.845093    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c79db4c74-j496b_037f095d-ebe7-44bd-8c85-0902026307ae/router/0.log"
May 28 11:06:02.047873 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:02.047843    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8pg7p_99ba1953-8c33-4ce7-81dd-ff524369264f/serve-healthcheck-canary/0.log"
May 28 11:06:04.748347 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:04.748316    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:06:04.752412 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:04.752382    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:06:36.133895 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:36.133856    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 11:06:36.134442 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:36.134296    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="alertmanager" containerID="cri-o://5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf" gracePeriod=120
May 28 11:06:36.134442 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:36.134371    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="kube-rbac-proxy-metric" containerID="cri-o://62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625" gracePeriod=120
May 28 11:06:36.134561 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:36.134406    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="prom-label-proxy" containerID="cri-o://0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343" gracePeriod=120
May 28 11:06:36.134561 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:36.134415    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="kube-rbac-proxy-web" containerID="cri-o://c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e" gracePeriod=120
May 28 11:06:36.134675 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:36.134462    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="kube-rbac-proxy" containerID="cri-o://d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3" gracePeriod=120
May 28 11:06:36.135075 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:36.134664    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="config-reloader" containerID="cri-o://0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd" gracePeriod=120
May 28 11:06:37.215435 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.215404    2552 generic.go:358] "Generic (PLEG): container finished" podID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerID="0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343" exitCode=0
May 28 11:06:37.215435 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.215432    2552 generic.go:358] "Generic (PLEG): container finished" podID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerID="d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3" exitCode=0
May 28 11:06:37.215435 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.215439    2552 generic.go:358] "Generic (PLEG): container finished" podID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerID="0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd" exitCode=0
May 28 11:06:37.215831 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.215448    2552 generic.go:358] "Generic (PLEG): container finished" podID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerID="5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf" exitCode=0
May 28 11:06:37.215831 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.215471    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c49803-48d5-4664-8cae-caaa0542a6d1","Type":"ContainerDied","Data":"0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343"}
May 28 11:06:37.215831 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.215502    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c49803-48d5-4664-8cae-caaa0542a6d1","Type":"ContainerDied","Data":"d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3"}
May 28 11:06:37.215831 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.215515    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c49803-48d5-4664-8cae-caaa0542a6d1","Type":"ContainerDied","Data":"0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd"}
May 28 11:06:37.215831 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.215525    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c49803-48d5-4664-8cae-caaa0542a6d1","Type":"ContainerDied","Data":"5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf"}
May 28 11:06:37.379758 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.379733    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:37.466203 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466103    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vglr9\" (UniqueName: \"kubernetes.io/projected/76c49803-48d5-4664-8cae-caaa0542a6d1-kube-api-access-vglr9\") pod \"76c49803-48d5-4664-8cae-caaa0542a6d1\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") "
May 28 11:06:37.466203 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466165    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c49803-48d5-4664-8cae-caaa0542a6d1-alertmanager-trusted-ca-bundle\") pod \"76c49803-48d5-4664-8cae-caaa0542a6d1\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") "
May 28 11:06:37.466203 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466187    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76c49803-48d5-4664-8cae-caaa0542a6d1-tls-assets\") pod \"76c49803-48d5-4664-8cae-caaa0542a6d1\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") "
May 28 11:06:37.466445 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466270    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"76c49803-48d5-4664-8cae-caaa0542a6d1\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") "
May 28 11:06:37.466445 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466300    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/76c49803-48d5-4664-8cae-caaa0542a6d1-alertmanager-main-db\") pod \"76c49803-48d5-4664-8cae-caaa0542a6d1\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") "
May 28 11:06:37.466445 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466327    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-config-volume\") pod \"76c49803-48d5-4664-8cae-caaa0542a6d1\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") "
May 28 11:06:37.466595 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466451    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76c49803-48d5-4664-8cae-caaa0542a6d1-config-out\") pod \"76c49803-48d5-4664-8cae-caaa0542a6d1\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") "
May 28 11:06:37.466595 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466520    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-web-config\") pod \"76c49803-48d5-4664-8cae-caaa0542a6d1\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") "
May 28 11:06:37.466595 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466575    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy-web\") pod \"76c49803-48d5-4664-8cae-caaa0542a6d1\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") "
May 28 11:06:37.466736 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466589    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c49803-48d5-4664-8cae-caaa0542a6d1-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "76c49803-48d5-4664-8cae-caaa0542a6d1" (UID: "76c49803-48d5-4664-8cae-caaa0542a6d1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 11:06:37.466736 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466605    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-main-tls\") pod \"76c49803-48d5-4664-8cae-caaa0542a6d1\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") "
May 28 11:06:37.466736 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466630    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c49803-48d5-4664-8cae-caaa0542a6d1-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "76c49803-48d5-4664-8cae-caaa0542a6d1" (UID: "76c49803-48d5-4664-8cae-caaa0542a6d1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 11:06:37.466736 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466660    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-cluster-tls-config\") pod \"76c49803-48d5-4664-8cae-caaa0542a6d1\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") "
May 28 11:06:37.466736 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466694    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy\") pod \"76c49803-48d5-4664-8cae-caaa0542a6d1\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") "
May 28 11:06:37.466736 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466727    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76c49803-48d5-4664-8cae-caaa0542a6d1-metrics-client-ca\") pod \"76c49803-48d5-4664-8cae-caaa0542a6d1\" (UID: \"76c49803-48d5-4664-8cae-caaa0542a6d1\") "
May 28 11:06:37.467043 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466974    2552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76c49803-48d5-4664-8cae-caaa0542a6d1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:06:37.467043 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.466994    2552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/76c49803-48d5-4664-8cae-caaa0542a6d1-alertmanager-main-db\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:06:37.467501 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.467465    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c49803-48d5-4664-8cae-caaa0542a6d1-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "76c49803-48d5-4664-8cae-caaa0542a6d1" (UID: "76c49803-48d5-4664-8cae-caaa0542a6d1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 11:06:37.469033 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.469003    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c49803-48d5-4664-8cae-caaa0542a6d1-kube-api-access-vglr9" (OuterVolumeSpecName: "kube-api-access-vglr9") pod "76c49803-48d5-4664-8cae-caaa0542a6d1" (UID: "76c49803-48d5-4664-8cae-caaa0542a6d1"). InnerVolumeSpecName "kube-api-access-vglr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:06:37.469386 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.469357    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c49803-48d5-4664-8cae-caaa0542a6d1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "76c49803-48d5-4664-8cae-caaa0542a6d1" (UID: "76c49803-48d5-4664-8cae-caaa0542a6d1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:06:37.469794 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.469638    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "76c49803-48d5-4664-8cae-caaa0542a6d1" (UID: "76c49803-48d5-4664-8cae-caaa0542a6d1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:06:37.469794 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.469760    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "76c49803-48d5-4664-8cae-caaa0542a6d1" (UID: "76c49803-48d5-4664-8cae-caaa0542a6d1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:06:37.470057 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.470028    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "76c49803-48d5-4664-8cae-caaa0542a6d1" (UID: "76c49803-48d5-4664-8cae-caaa0542a6d1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:06:37.470323 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.470297    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c49803-48d5-4664-8cae-caaa0542a6d1-config-out" (OuterVolumeSpecName: "config-out") pod "76c49803-48d5-4664-8cae-caaa0542a6d1" (UID: "76c49803-48d5-4664-8cae-caaa0542a6d1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 11:06:37.470507 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.470489    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-config-volume" (OuterVolumeSpecName: "config-volume") pod "76c49803-48d5-4664-8cae-caaa0542a6d1" (UID: "76c49803-48d5-4664-8cae-caaa0542a6d1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:06:37.471024 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.471005    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "76c49803-48d5-4664-8cae-caaa0542a6d1" (UID: "76c49803-48d5-4664-8cae-caaa0542a6d1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:06:37.474501 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.474357    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "76c49803-48d5-4664-8cae-caaa0542a6d1" (UID: "76c49803-48d5-4664-8cae-caaa0542a6d1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:06:37.480189 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.480168    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-web-config" (OuterVolumeSpecName: "web-config") pod "76c49803-48d5-4664-8cae-caaa0542a6d1" (UID: "76c49803-48d5-4664-8cae-caaa0542a6d1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:06:37.568105 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.568066    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76c49803-48d5-4664-8cae-caaa0542a6d1-metrics-client-ca\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:06:37.568105 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.568097    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vglr9\" (UniqueName: \"kubernetes.io/projected/76c49803-48d5-4664-8cae-caaa0542a6d1-kube-api-access-vglr9\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:06:37.568105 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.568106    2552 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76c49803-48d5-4664-8cae-caaa0542a6d1-tls-assets\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:06:37.568105 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.568116    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:06:37.568420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.568126    2552 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-config-volume\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:06:37.568420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.568158    2552 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76c49803-48d5-4664-8cae-caaa0542a6d1-config-out\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:06:37.568420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.568170    2552 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-web-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:06:37.568420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.568190    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:06:37.568420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.568201    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-main-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:06:37.568420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.568212    2552 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-cluster-tls-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:06:37.568420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:37.568221    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76c49803-48d5-4664-8cae-caaa0542a6d1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:06:38.220909 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.220877    2552 generic.go:358] "Generic (PLEG): container finished" podID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerID="62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625" exitCode=0
May 28 11:06:38.220909 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.220902    2552 generic.go:358] "Generic (PLEG): container finished" podID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerID="c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e" exitCode=0
May 28 11:06:38.221319 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.220959    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c49803-48d5-4664-8cae-caaa0542a6d1","Type":"ContainerDied","Data":"62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625"}
May 28 11:06:38.221319 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.220977    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.221319 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.221000    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c49803-48d5-4664-8cae-caaa0542a6d1","Type":"ContainerDied","Data":"c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e"}
May 28 11:06:38.221319 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.221012    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76c49803-48d5-4664-8cae-caaa0542a6d1","Type":"ContainerDied","Data":"39fda053d7819140d789268045ea29d0f3d52f63074749d6cd2ea39e73d0100c"}
May 28 11:06:38.221319 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.221026    2552 scope.go:117] "RemoveContainer" containerID="0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343"
May 28 11:06:38.228806 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.228784    2552 scope.go:117] "RemoveContainer" containerID="62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625"
May 28 11:06:38.235312 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.235292    2552 scope.go:117] "RemoveContainer" containerID="d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3"
May 28 11:06:38.241543 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.241517    2552 scope.go:117] "RemoveContainer" containerID="c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e"
May 28 11:06:38.243552 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.243532    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 11:06:38.248353 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.248334    2552 scope.go:117] "RemoveContainer" containerID="0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd"
May 28 11:06:38.248922 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.248900    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 11:06:38.255003 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.254842    2552 scope.go:117] "RemoveContainer" containerID="5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf"
May 28 11:06:38.260991 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.260973    2552 scope.go:117] "RemoveContainer" containerID="a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd"
May 28 11:06:38.267283 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.267262    2552 scope.go:117] "RemoveContainer" containerID="0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343"
May 28 11:06:38.267540 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:06:38.267519    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343\": container with ID starting with 0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343 not found: ID does not exist" containerID="0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343"
May 28 11:06:38.267606 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.267549    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343"} err="failed to get container status \"0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343\": rpc error: code = NotFound desc = could not find container \"0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343\": container with ID starting with 0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343 not found: ID does not exist"
May 28 11:06:38.267606 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.267569    2552 scope.go:117] "RemoveContainer" containerID="62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625"
May 28 11:06:38.267759 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:06:38.267745    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625\": container with ID starting with 62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625 not found: ID does not exist" containerID="62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625"
May 28 11:06:38.267833 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.267764    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625"} err="failed to get container status \"62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625\": rpc error: code = NotFound desc = could not find container \"62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625\": container with ID starting with 62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625 not found: ID does not exist"
May 28 11:06:38.267833 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.267776    2552 scope.go:117] "RemoveContainer" containerID="d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3"
May 28 11:06:38.267960 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:06:38.267944    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3\": container with ID starting with d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3 not found: ID does not exist" containerID="d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3"
May 28 11:06:38.268002 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.267962    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3"} err="failed to get container status \"d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3\": rpc error: code = NotFound desc = could not find container \"d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3\": container with ID starting with d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3 not found: ID does not exist"
May 28 11:06:38.268002 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.267981    2552 scope.go:117] "RemoveContainer" containerID="c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e"
May 28 11:06:38.268204 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:06:38.268187    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e\": container with ID starting with c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e not found: ID does not exist" containerID="c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e"
May 28 11:06:38.268263 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.268209    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e"} err="failed to get container status \"c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e\": rpc error: code = NotFound desc = could not find container \"c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e\": container with ID starting with c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e not found: ID does not exist"
May 28 11:06:38.268263 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.268225    2552 scope.go:117] "RemoveContainer" containerID="0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd"
May 28 11:06:38.268447 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:06:38.268429    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd\": container with ID starting with 0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd not found: ID does not exist" containerID="0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd"
May 28 11:06:38.268487 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.268451    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd"} err="failed to get container status \"0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd\": rpc error: code = NotFound desc = could not find container \"0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd\": container with ID starting with 0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd not found: ID does not exist"
May 28 11:06:38.268487 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.268464    2552 scope.go:117] "RemoveContainer" containerID="5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf"
May 28 11:06:38.268720 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:06:38.268693    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf\": container with ID starting with 5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf not found: ID does not exist" containerID="5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf"
May 28 11:06:38.268787 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.268716    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf"} err="failed to get container status \"5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf\": rpc error: code = NotFound desc = could not find container \"5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf\": container with ID starting with 5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf not found: ID does not exist"
May 28 11:06:38.268787 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.268741    2552 scope.go:117] "RemoveContainer" containerID="a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd"
May 28 11:06:38.268975 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:06:38.268960    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd\": container with ID starting with a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd not found: ID does not exist" containerID="a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd"
May 28 11:06:38.269013 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.268978    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd"} err="failed to get container status \"a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd\": rpc error: code = NotFound desc = could not find container \"a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd\": container with ID starting with a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd not found: ID does not exist"
May 28 11:06:38.269013 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.268993    2552 scope.go:117] "RemoveContainer" containerID="0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343"
May 28 11:06:38.269231 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.269211    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343"} err="failed to get container status \"0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343\": rpc error: code = NotFound desc = could not find container \"0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343\": container with ID starting with 0e6f9d183046bc128c959618a47be6b9b6cbdbd9b7e498f9f930082238458343 not found: ID does not exist"
May 28 11:06:38.269269 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.269232    2552 scope.go:117] "RemoveContainer" containerID="62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625"
May 28 11:06:38.269463 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.269444    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625"} err="failed to get container status \"62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625\": rpc error: code = NotFound desc = could not find container \"62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625\": container with ID starting with 62f0e226390d699f170f2d32e598e2d8b31494492314500a21a31448dbdc4625 not found: ID does not exist"
May 28 11:06:38.269526 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.269466    2552 scope.go:117] "RemoveContainer" containerID="d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3"
May 28 11:06:38.269697 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.269680    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3"} err="failed to get container status \"d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3\": rpc error: code = NotFound desc = could not find container \"d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3\": container with ID starting with d3a4afdc5f9e5cc1df56aa6226b328a93f8a0d39072b44d6bfd066f94bd5d2d3 not found: ID does not exist"
May 28 11:06:38.269750 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.269698    2552 scope.go:117] "RemoveContainer" containerID="c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e"
May 28 11:06:38.269892 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.269877    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e"} err="failed to get container status \"c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e\": rpc error: code = NotFound desc = could not find container \"c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e\": container with ID starting with c27ee82929d50ad496dd79948b5b79e013a7c87e49f0bcba2264b0ac46b46c1e not found: ID does not exist"
May 28 11:06:38.269932 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.269893    2552 scope.go:117] "RemoveContainer" containerID="0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd"
May 28 11:06:38.270076 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.270056    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd"} err="failed to get container status \"0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd\": rpc error: code = NotFound desc = could not find container \"0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd\": container with ID starting with 0b25977ea91f628245155512dff45aabf0ccda0a484b482a81ee9d2925b3e1dd not found: ID does not exist"
May 28 11:06:38.270119 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.270077    2552 scope.go:117] "RemoveContainer" containerID="5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf"
May 28 11:06:38.270308 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.270291    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf"} err="failed to get container status \"5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf\": rpc error: code = NotFound desc = could not find container \"5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf\": container with ID starting with 5efd6c345a5b2542035ec4245006f8f26f479999c2ee34bd2da9c686d59e73cf not found: ID does not exist"
May 28 11:06:38.270354 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.270308    2552 scope.go:117] "RemoveContainer" containerID="a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd"
May 28 11:06:38.270504 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.270487    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd"} err="failed to get container status \"a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd\": rpc error: code = NotFound desc = could not find container \"a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd\": container with ID starting with a01355bf679ffc792d8a83d449e9bbbcc6da808aa0775e0916aef769a9436bbd not found: ID does not exist"
May 28 11:06:38.278402 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278378    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 11:06:38.278636 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278624    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="config-reloader"
May 28 11:06:38.278677 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278638    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="config-reloader"
May 28 11:06:38.278677 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278650    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="init-config-reloader"
May 28 11:06:38.278677 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278655    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="init-config-reloader"
May 28 11:06:38.278677 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278661    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="alertmanager"
May 28 11:06:38.278677 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278667    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="alertmanager"
May 28 11:06:38.278677 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278674    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="kube-rbac-proxy-web"
May 28 11:06:38.278677 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278679    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="kube-rbac-proxy-web"
May 28 11:06:38.278884 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278691    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="kube-rbac-proxy"
May 28 11:06:38.278884 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278697    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="kube-rbac-proxy"
May 28 11:06:38.278884 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278703    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="prom-label-proxy"
May 28 11:06:38.278884 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278709    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="prom-label-proxy"
May 28 11:06:38.278884 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278716    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="kube-rbac-proxy-metric"
May 28 11:06:38.278884 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278721    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="kube-rbac-proxy-metric"
May 28 11:06:38.278884 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278767    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="kube-rbac-proxy"
May 28 11:06:38.278884 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278776    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="kube-rbac-proxy-metric"
May 28 11:06:38.278884 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278783    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="alertmanager"
May 28 11:06:38.278884 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278788    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="config-reloader"
May 28 11:06:38.278884 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278793    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="kube-rbac-proxy-web"
May 28 11:06:38.278884 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.278800    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" containerName="prom-label-proxy"
May 28 11:06:38.283657 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.282164    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.284436 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.284414    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 28 11:06:38.284436 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.284414    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 28 11:06:38.284565 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.284454    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 28 11:06:38.284565 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.284460    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fc62h\""
May 28 11:06:38.284763 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.284744    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 28 11:06:38.284819 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.284754    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 28 11:06:38.285311 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.285292    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 28 11:06:38.285311 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.285305    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 28 11:06:38.285602 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.285584    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 28 11:06:38.289713 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.289693    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 28 11:06:38.295028 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.295005    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 11:06:38.374025 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.373987    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.374217 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.374034    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b79jr\" (UniqueName: \"kubernetes.io/projected/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-kube-api-access-b79jr\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.374217 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.374061    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.374217 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.374087    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.374217 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.374109    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-config-volume\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.374217 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.374123    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.374217 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.374194    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.374418 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.374221    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.374418 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.374239    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.374418 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.374259    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-config-out\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.374418 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.374278    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.374418 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.374324    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.374418 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.374346    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-web-config\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.475548 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.475513    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-config-out\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.475717 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.475557    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.475717 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.475581    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.475717 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.475604    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-web-config\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.475717 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.475625    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.475717 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.475651    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b79jr\" (UniqueName: \"kubernetes.io/projected/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-kube-api-access-b79jr\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.475717 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.475666    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.475717 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.475682    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.475717 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.475699    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-config-volume\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.476076 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.475998    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.476163 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.476093    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.476236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.476214    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.476297 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.476255    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.476353 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.476285    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.476947 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.476918    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.477067 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.476955    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.478837 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.478804    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-config-out\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.478939 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.478878    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-config-volume\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.478939 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.478894    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.478939 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.478877    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-web-config\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.479354 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.479330    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.479473 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.479458    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.479535 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.479470    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.479535 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.479515    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.480890 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.480863    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.482855 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.482836    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b79jr\" (UniqueName: \"kubernetes.io/projected/b57bf21e-ab6e-496e-b9ea-5589436e0ed9-kube-api-access-b79jr\") pod \"alertmanager-main-0\" (UID: \"b57bf21e-ab6e-496e-b9ea-5589436e0ed9\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.516515 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.516485    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76c49803-48d5-4664-8cae-caaa0542a6d1" path="/var/lib/kubelet/pods/76c49803-48d5-4664-8cae-caaa0542a6d1/volumes"
May 28 11:06:38.592884 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.592856    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 11:06:38.725857 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:38.725748    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 11:06:38.729586 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:06:38.729559    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57bf21e_ab6e_496e_b9ea_5589436e0ed9.slice/crio-d377e69ee8cf73bd23b88087105daf2f5683de3abe170f31e79dcf9c9f39f1b2 WatchSource:0}: Error finding container d377e69ee8cf73bd23b88087105daf2f5683de3abe170f31e79dcf9c9f39f1b2: Status 404 returned error can't find the container with id d377e69ee8cf73bd23b88087105daf2f5683de3abe170f31e79dcf9c9f39f1b2
May 28 11:06:39.225622 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:39.225587    2552 generic.go:358] "Generic (PLEG): container finished" podID="b57bf21e-ab6e-496e-b9ea-5589436e0ed9" containerID="42eed0c4bd4da4b093e231e9568e60dccd52d802d9ad0bffd6561030e3f8b8c2" exitCode=0
May 28 11:06:39.226054 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:39.225675    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b57bf21e-ab6e-496e-b9ea-5589436e0ed9","Type":"ContainerDied","Data":"42eed0c4bd4da4b093e231e9568e60dccd52d802d9ad0bffd6561030e3f8b8c2"}
May 28 11:06:39.226054 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:39.225721    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b57bf21e-ab6e-496e-b9ea-5589436e0ed9","Type":"ContainerStarted","Data":"d377e69ee8cf73bd23b88087105daf2f5683de3abe170f31e79dcf9c9f39f1b2"}
May 28 11:06:40.149727 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.149695    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"]
May 28 11:06:40.152334 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.152311    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.154957 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.154679    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 28 11:06:40.154957 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.154717    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vm2rs\""
May 28 11:06:40.154957 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.154739    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 28 11:06:40.155202 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.155003    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 28 11:06:40.155253 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.155211    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 28 11:06:40.155364 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.155347    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 28 11:06:40.160945 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.160925    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 28 11:06:40.165664 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.165639    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"]
May 28 11:06:40.233778 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.233740    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b57bf21e-ab6e-496e-b9ea-5589436e0ed9","Type":"ContainerStarted","Data":"4a99feb47154c0d5dae3e07818f0ce8eb3b3a81b119c2f0621db9c5da9a61e7e"}
May 28 11:06:40.233778 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.233782    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b57bf21e-ab6e-496e-b9ea-5589436e0ed9","Type":"ContainerStarted","Data":"40689153a38df4c65555201d733f74cd244894c79f8f1bcfc32e33999c99bee0"}
May 28 11:06:40.234194 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.233797    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b57bf21e-ab6e-496e-b9ea-5589436e0ed9","Type":"ContainerStarted","Data":"d714cf3adbd59c329d19904c6b830f02da2705a085b3aeb89c93f76f3664f2e8"}
May 28 11:06:40.234194 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.233811    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b57bf21e-ab6e-496e-b9ea-5589436e0ed9","Type":"ContainerStarted","Data":"1b120efa6fb96a84188deb7ebc38f8b2b30b9aef1ed6ea17b36e71e2231e7238"}
May 28 11:06:40.234194 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.233825    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b57bf21e-ab6e-496e-b9ea-5589436e0ed9","Type":"ContainerStarted","Data":"55313300e326bf47f8cc0b469dd45aa55cd14c48ac17bbdc35032ee98ba3ffa4"}
May 28 11:06:40.234194 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.233838    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b57bf21e-ab6e-496e-b9ea-5589436e0ed9","Type":"ContainerStarted","Data":"dcc42efa44ba78edddaefc4777bdc2e4eb604ea2782ac4db2f2cb7caa9dbfb64"}
May 28 11:06:40.259970 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.259907    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.259877988 podStartE2EDuration="2.259877988s" podCreationTimestamp="2026-05-28 11:06:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 11:06:40.257633088 +0000 UTC m=+216.296730067" watchObservedRunningTime="2026-05-28 11:06:40.259877988 +0000 UTC m=+216.298974978"
May 28 11:06:40.291006 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.290970    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-secret-telemeter-client\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.291176 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.291017    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-serving-certs-ca-bundle\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.291176 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.291068    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.291176 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.291115    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc4zh\" (UniqueName: \"kubernetes.io/projected/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-kube-api-access-xc4zh\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.291288 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.291185    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-telemeter-client-tls\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.291288 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.291202    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-federate-client-tls\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.291288 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.291227    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-metrics-client-ca\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.291288 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.291247    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.391787 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.391754    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.392002 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.391933    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xc4zh\" (UniqueName: \"kubernetes.io/projected/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-kube-api-access-xc4zh\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.392190 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.392172    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-telemeter-client-tls\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.392275 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.392210    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-federate-client-tls\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.392444 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.392419    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-metrics-client-ca\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.392511 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.392465    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.392578 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.392513    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-secret-telemeter-client\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.392578 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.392543    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-serving-certs-ca-bundle\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.393322 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.393298    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-metrics-client-ca\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.393431 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.393298    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-serving-certs-ca-bundle\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.393986 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.393962    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.394825 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.394801    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.394905 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.394857    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-federate-client-tls\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.394972 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.394958    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-telemeter-client-tls\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.395324 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.395305    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-secret-telemeter-client\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.401814 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.400504    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc4zh\" (UniqueName: \"kubernetes.io/projected/8179a300-6b5b-428e-aa0b-c0e4beafd3d5-kube-api-access-xc4zh\") pod \"telemeter-client-85cfd9cc64-kk8pm\" (UID: \"8179a300-6b5b-428e-aa0b-c0e4beafd3d5\") " pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.466512 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.466478    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"
May 28 11:06:40.616350 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:40.616315    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm"]
May 28 11:06:40.619407 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:06:40.619376    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8179a300_6b5b_428e_aa0b_c0e4beafd3d5.slice/crio-db44cae1a22a0c4393fc5dc62d7a72d02fb79db9bbe43bdbc43d159289cb368f WatchSource:0}: Error finding container db44cae1a22a0c4393fc5dc62d7a72d02fb79db9bbe43bdbc43d159289cb368f: Status 404 returned error can't find the container with id db44cae1a22a0c4393fc5dc62d7a72d02fb79db9bbe43bdbc43d159289cb368f
May 28 11:06:41.240550 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:41.240506    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm" event={"ID":"8179a300-6b5b-428e-aa0b-c0e4beafd3d5","Type":"ContainerStarted","Data":"db44cae1a22a0c4393fc5dc62d7a72d02fb79db9bbe43bdbc43d159289cb368f"}
May 28 11:06:43.247772 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:43.247738    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm" event={"ID":"8179a300-6b5b-428e-aa0b-c0e4beafd3d5","Type":"ContainerStarted","Data":"6e9a7840709d7f52df9a5304d5ab180097ccad089f12a3c5795f464d5ba7c24f"}
May 28 11:06:43.247772 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:43.247775    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm" event={"ID":"8179a300-6b5b-428e-aa0b-c0e4beafd3d5","Type":"ContainerStarted","Data":"603875bcae55e347e8e57d945c184c6df2d6518242a58485fed21ba9353f98be"}
May 28 11:06:43.248216 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:43.247786    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm" event={"ID":"8179a300-6b5b-428e-aa0b-c0e4beafd3d5","Type":"ContainerStarted","Data":"2c59bef30c0cf46afa944555cae55f38d6286864b1af37d15b3870d5f5e6feb6"}
May 28 11:06:43.269324 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:43.269274    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-85cfd9cc64-kk8pm" podStartSLOduration=1.545931399 podStartE2EDuration="3.269259543s" podCreationTimestamp="2026-05-28 11:06:40 +0000 UTC" firstStartedPulling="2026-05-28 11:06:40.621094598 +0000 UTC m=+216.660191555" lastFinishedPulling="2026-05-28 11:06:42.344422738 +0000 UTC m=+218.383519699" observedRunningTime="2026-05-28 11:06:43.26780012 +0000 UTC m=+219.306897098" watchObservedRunningTime="2026-05-28 11:06:43.269259543 +0000 UTC m=+219.308356521"
May 28 11:06:44.229280 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.229238    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-bff887bb5-6hswq"]
May 28 11:06:44.232457 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.232403    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.248932 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.248904    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bff887bb5-6hswq"]
May 28 11:06:44.329235 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.329187    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41941f76-780c-485d-8a77-140318ee6e89-console-serving-cert\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.329409 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.329295    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-console-config\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.329409 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.329403    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6782x\" (UniqueName: \"kubernetes.io/projected/41941f76-780c-485d-8a77-140318ee6e89-kube-api-access-6782x\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.329613 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.329592    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-trusted-ca-bundle\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.329763 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.329743    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41941f76-780c-485d-8a77-140318ee6e89-console-oauth-config\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.329819 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.329777    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-oauth-serving-cert\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.329853 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.329842    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-service-ca\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.430966 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.430921    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-console-config\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.431153 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.431080    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6782x\" (UniqueName: \"kubernetes.io/projected/41941f76-780c-485d-8a77-140318ee6e89-kube-api-access-6782x\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.431153 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.431149    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-trusted-ca-bundle\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.431238 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.431197    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41941f76-780c-485d-8a77-140318ee6e89-console-oauth-config\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.431238 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.431218    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-oauth-serving-cert\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.431331 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.431260    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-service-ca\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.431331 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.431306    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41941f76-780c-485d-8a77-140318ee6e89-console-serving-cert\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.431766 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.431737    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-console-config\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.431966 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.431939    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-oauth-serving-cert\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.432056 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.431980    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-trusted-ca-bundle\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.432056 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.432036    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-service-ca\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.433860 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.433815    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41941f76-780c-485d-8a77-140318ee6e89-console-serving-cert\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.434186 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.434162    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41941f76-780c-485d-8a77-140318ee6e89-console-oauth-config\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.439632 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.439608    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6782x\" (UniqueName: \"kubernetes.io/projected/41941f76-780c-485d-8a77-140318ee6e89-kube-api-access-6782x\") pod \"console-bff887bb5-6hswq\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") " pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.543753 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.543651    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:44.669420 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:44.669384    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bff887bb5-6hswq"]
May 28 11:06:44.672174 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:06:44.672142    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41941f76_780c_485d_8a77_140318ee6e89.slice/crio-4fa1ccc67d6b5add9223cd3107d96010b30928e1616caf47ae1702655653e6b4 WatchSource:0}: Error finding container 4fa1ccc67d6b5add9223cd3107d96010b30928e1616caf47ae1702655653e6b4: Status 404 returned error can't find the container with id 4fa1ccc67d6b5add9223cd3107d96010b30928e1616caf47ae1702655653e6b4
May 28 11:06:45.257857 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:45.257820    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bff887bb5-6hswq" event={"ID":"41941f76-780c-485d-8a77-140318ee6e89","Type":"ContainerStarted","Data":"e9746a9f248754d3355ad074542f11d4b5984c3b3de06a52ad3b133a5e4b9848"}
May 28 11:06:45.257857 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:45.257857    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bff887bb5-6hswq" event={"ID":"41941f76-780c-485d-8a77-140318ee6e89","Type":"ContainerStarted","Data":"4fa1ccc67d6b5add9223cd3107d96010b30928e1616caf47ae1702655653e6b4"}
May 28 11:06:45.274894 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:45.274844    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-bff887bb5-6hswq" podStartSLOduration=1.274831146 podStartE2EDuration="1.274831146s" podCreationTimestamp="2026-05-28 11:06:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 11:06:45.273160948 +0000 UTC m=+221.312257927" watchObservedRunningTime="2026-05-28 11:06:45.274831146 +0000 UTC m=+221.313928125"
May 28 11:06:54.543965 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:54.543916    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:54.543965 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:54.543965    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:54.548839 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:54.548812    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:55.295355 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:55.295323    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:06:55.341280 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:06:55.341241    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66d9b758d6-l5r6m"]
May 28 11:07:20.363860 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.363770    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-66d9b758d6-l5r6m" podUID="5487a12f-8342-410a-8c8d-523fadb4b31b" containerName="console" containerID="cri-o://b483950f076472eb61cf104d4e7509bd44182126c987ef2368f15d96ad7b7912" gracePeriod=15
May 28 11:07:20.602427 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.602398    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-66d9b758d6-l5r6m_5487a12f-8342-410a-8c8d-523fadb4b31b/console/0.log"
May 28 11:07:20.602535 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.602459    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:07:20.608038 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.608015    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-console-config\") pod \"5487a12f-8342-410a-8c8d-523fadb4b31b\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") "
May 28 11:07:20.608169 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.608058    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5487a12f-8342-410a-8c8d-523fadb4b31b-console-oauth-config\") pod \"5487a12f-8342-410a-8c8d-523fadb4b31b\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") "
May 28 11:07:20.608169 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.608091    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-trusted-ca-bundle\") pod \"5487a12f-8342-410a-8c8d-523fadb4b31b\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") "
May 28 11:07:20.608169 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.608142    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5487a12f-8342-410a-8c8d-523fadb4b31b-console-serving-cert\") pod \"5487a12f-8342-410a-8c8d-523fadb4b31b\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") "
May 28 11:07:20.608335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.608173    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-oauth-serving-cert\") pod \"5487a12f-8342-410a-8c8d-523fadb4b31b\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") "
May 28 11:07:20.608335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.608201    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-service-ca\") pod \"5487a12f-8342-410a-8c8d-523fadb4b31b\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") "
May 28 11:07:20.608335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.608246    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5wwz\" (UniqueName: \"kubernetes.io/projected/5487a12f-8342-410a-8c8d-523fadb4b31b-kube-api-access-d5wwz\") pod \"5487a12f-8342-410a-8c8d-523fadb4b31b\" (UID: \"5487a12f-8342-410a-8c8d-523fadb4b31b\") "
May 28 11:07:20.608469 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.608414    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-console-config" (OuterVolumeSpecName: "console-config") pod "5487a12f-8342-410a-8c8d-523fadb4b31b" (UID: "5487a12f-8342-410a-8c8d-523fadb4b31b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 11:07:20.608610 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.608587    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5487a12f-8342-410a-8c8d-523fadb4b31b" (UID: "5487a12f-8342-410a-8c8d-523fadb4b31b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 11:07:20.608682 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.608584    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-service-ca" (OuterVolumeSpecName: "service-ca") pod "5487a12f-8342-410a-8c8d-523fadb4b31b" (UID: "5487a12f-8342-410a-8c8d-523fadb4b31b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 11:07:20.608682 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.608618    2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-console-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:07:20.608682 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.608619    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5487a12f-8342-410a-8c8d-523fadb4b31b" (UID: "5487a12f-8342-410a-8c8d-523fadb4b31b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 11:07:20.610454 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.610425    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5487a12f-8342-410a-8c8d-523fadb4b31b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5487a12f-8342-410a-8c8d-523fadb4b31b" (UID: "5487a12f-8342-410a-8c8d-523fadb4b31b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:07:20.610454 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.610444    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5487a12f-8342-410a-8c8d-523fadb4b31b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5487a12f-8342-410a-8c8d-523fadb4b31b" (UID: "5487a12f-8342-410a-8c8d-523fadb4b31b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:07:20.610621 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.610520    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5487a12f-8342-410a-8c8d-523fadb4b31b-kube-api-access-d5wwz" (OuterVolumeSpecName: "kube-api-access-d5wwz") pod "5487a12f-8342-410a-8c8d-523fadb4b31b" (UID: "5487a12f-8342-410a-8c8d-523fadb4b31b"). InnerVolumeSpecName "kube-api-access-d5wwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:07:20.709535 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.709502    2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5487a12f-8342-410a-8c8d-523fadb4b31b-console-oauth-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:07:20.709535 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.709528    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-trusted-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:07:20.709535 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.709539    2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5487a12f-8342-410a-8c8d-523fadb4b31b-console-serving-cert\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:07:20.709770 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.709548    2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-oauth-serving-cert\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:07:20.709770 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.709558    2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5487a12f-8342-410a-8c8d-523fadb4b31b-service-ca\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:07:20.709770 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:20.709567    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5wwz\" (UniqueName: \"kubernetes.io/projected/5487a12f-8342-410a-8c8d-523fadb4b31b-kube-api-access-d5wwz\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:07:21.368056 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:21.368029    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-66d9b758d6-l5r6m_5487a12f-8342-410a-8c8d-523fadb4b31b/console/0.log"
May 28 11:07:21.368478 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:21.368073    2552 generic.go:358] "Generic (PLEG): container finished" podID="5487a12f-8342-410a-8c8d-523fadb4b31b" containerID="b483950f076472eb61cf104d4e7509bd44182126c987ef2368f15d96ad7b7912" exitCode=2
May 28 11:07:21.368478 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:21.368143    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d9b758d6-l5r6m"
May 28 11:07:21.368478 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:21.368161    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d9b758d6-l5r6m" event={"ID":"5487a12f-8342-410a-8c8d-523fadb4b31b","Type":"ContainerDied","Data":"b483950f076472eb61cf104d4e7509bd44182126c987ef2368f15d96ad7b7912"}
May 28 11:07:21.368478 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:21.368200    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d9b758d6-l5r6m" event={"ID":"5487a12f-8342-410a-8c8d-523fadb4b31b","Type":"ContainerDied","Data":"3bdb05abbc0fe4523602b8b7ab90cd7618c875a848cb8e76082094783ba8dc77"}
May 28 11:07:21.368478 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:21.368215    2552 scope.go:117] "RemoveContainer" containerID="b483950f076472eb61cf104d4e7509bd44182126c987ef2368f15d96ad7b7912"
May 28 11:07:21.376597 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:21.376581    2552 scope.go:117] "RemoveContainer" containerID="b483950f076472eb61cf104d4e7509bd44182126c987ef2368f15d96ad7b7912"
May 28 11:07:21.376818 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:07:21.376800    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b483950f076472eb61cf104d4e7509bd44182126c987ef2368f15d96ad7b7912\": container with ID starting with b483950f076472eb61cf104d4e7509bd44182126c987ef2368f15d96ad7b7912 not found: ID does not exist" containerID="b483950f076472eb61cf104d4e7509bd44182126c987ef2368f15d96ad7b7912"
May 28 11:07:21.376868 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:21.376826    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b483950f076472eb61cf104d4e7509bd44182126c987ef2368f15d96ad7b7912"} err="failed to get container status \"b483950f076472eb61cf104d4e7509bd44182126c987ef2368f15d96ad7b7912\": rpc error: code = NotFound desc = could not find container \"b483950f076472eb61cf104d4e7509bd44182126c987ef2368f15d96ad7b7912\": container with ID starting with b483950f076472eb61cf104d4e7509bd44182126c987ef2368f15d96ad7b7912 not found: ID does not exist"
May 28 11:07:21.388204 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:21.388149    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66d9b758d6-l5r6m"]
May 28 11:07:21.392296 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:21.392271    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66d9b758d6-l5r6m"]
May 28 11:07:22.517348 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:07:22.517311    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5487a12f-8342-410a-8c8d-523fadb4b31b" path="/var/lib/kubelet/pods/5487a12f-8342-410a-8c8d-523fadb4b31b/volumes"
May 28 11:08:04.397257 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:04.397230    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:08:04.401184 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:04.401162    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:08:04.409903 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:04.409881    2552 kubelet.go:1628] "Image garbage collection succeeded"
May 28 11:08:07.321538 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:07.321501    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bff887bb5-6hswq"]
May 28 11:08:28.994947 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:28.994908    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-4znlc"]
May 28 11:08:28.995381 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:28.995287    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5487a12f-8342-410a-8c8d-523fadb4b31b" containerName="console"
May 28 11:08:28.995381 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:28.995300    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5487a12f-8342-410a-8c8d-523fadb4b31b" containerName="console"
May 28 11:08:28.995381 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:28.995355    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="5487a12f-8342-410a-8c8d-523fadb4b31b" containerName="console"
May 28 11:08:28.998683 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:28.998666    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4znlc"
May 28 11:08:29.000812 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:29.000792    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 28 11:08:29.004560 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:29.004534    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4znlc"]
May 28 11:08:29.070356 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:29.070311    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e5305c2-8214-4b76-9e9b-56fd284dbe37-original-pull-secret\") pod \"global-pull-secret-syncer-4znlc\" (UID: \"4e5305c2-8214-4b76-9e9b-56fd284dbe37\") " pod="kube-system/global-pull-secret-syncer-4znlc"
May 28 11:08:29.070531 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:29.070365    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4e5305c2-8214-4b76-9e9b-56fd284dbe37-kubelet-config\") pod \"global-pull-secret-syncer-4znlc\" (UID: \"4e5305c2-8214-4b76-9e9b-56fd284dbe37\") " pod="kube-system/global-pull-secret-syncer-4znlc"
May 28 11:08:29.070531 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:29.070458    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4e5305c2-8214-4b76-9e9b-56fd284dbe37-dbus\") pod \"global-pull-secret-syncer-4znlc\" (UID: \"4e5305c2-8214-4b76-9e9b-56fd284dbe37\") " pod="kube-system/global-pull-secret-syncer-4znlc"
May 28 11:08:29.171616 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:29.171580    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4e5305c2-8214-4b76-9e9b-56fd284dbe37-dbus\") pod \"global-pull-secret-syncer-4znlc\" (UID: \"4e5305c2-8214-4b76-9e9b-56fd284dbe37\") " pod="kube-system/global-pull-secret-syncer-4znlc"
May 28 11:08:29.171808 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:29.171631    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e5305c2-8214-4b76-9e9b-56fd284dbe37-original-pull-secret\") pod \"global-pull-secret-syncer-4znlc\" (UID: \"4e5305c2-8214-4b76-9e9b-56fd284dbe37\") " pod="kube-system/global-pull-secret-syncer-4znlc"
May 28 11:08:29.171808 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:29.171665    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4e5305c2-8214-4b76-9e9b-56fd284dbe37-kubelet-config\") pod \"global-pull-secret-syncer-4znlc\" (UID: \"4e5305c2-8214-4b76-9e9b-56fd284dbe37\") " pod="kube-system/global-pull-secret-syncer-4znlc"
May 28 11:08:29.171808 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:29.171745    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4e5305c2-8214-4b76-9e9b-56fd284dbe37-kubelet-config\") pod \"global-pull-secret-syncer-4znlc\" (UID: \"4e5305c2-8214-4b76-9e9b-56fd284dbe37\") " pod="kube-system/global-pull-secret-syncer-4znlc"
May 28 11:08:29.171808 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:29.171798    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4e5305c2-8214-4b76-9e9b-56fd284dbe37-dbus\") pod \"global-pull-secret-syncer-4znlc\" (UID: \"4e5305c2-8214-4b76-9e9b-56fd284dbe37\") " pod="kube-system/global-pull-secret-syncer-4znlc"
May 28 11:08:29.173867 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:29.173851    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4e5305c2-8214-4b76-9e9b-56fd284dbe37-original-pull-secret\") pod \"global-pull-secret-syncer-4znlc\" (UID: \"4e5305c2-8214-4b76-9e9b-56fd284dbe37\") " pod="kube-system/global-pull-secret-syncer-4znlc"
May 28 11:08:29.309061 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:29.308965    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4znlc"
May 28 11:08:29.434755 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:29.434718    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4znlc"]
May 28 11:08:29.437504 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:08:29.437476    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5305c2_8214_4b76_9e9b_56fd284dbe37.slice/crio-55dad5ccb7b7afc59cedef13ca0dd3d5904c11406d20c0c857d3273ee69768ad WatchSource:0}: Error finding container 55dad5ccb7b7afc59cedef13ca0dd3d5904c11406d20c0c857d3273ee69768ad: Status 404 returned error can't find the container with id 55dad5ccb7b7afc59cedef13ca0dd3d5904c11406d20c0c857d3273ee69768ad
May 28 11:08:29.439316 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:29.439294    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 11:08:29.567205 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:29.567100    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4znlc" event={"ID":"4e5305c2-8214-4b76-9e9b-56fd284dbe37","Type":"ContainerStarted","Data":"55dad5ccb7b7afc59cedef13ca0dd3d5904c11406d20c0c857d3273ee69768ad"}
May 28 11:08:32.342254 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:32.342183    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-bff887bb5-6hswq" podUID="41941f76-780c-485d-8a77-140318ee6e89" containerName="console" containerID="cri-o://e9746a9f248754d3355ad074542f11d4b5984c3b3de06a52ad3b133a5e4b9848" gracePeriod=15
May 28 11:08:32.582914 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:32.582872    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-bff887bb5-6hswq_41941f76-780c-485d-8a77-140318ee6e89/console/0.log"
May 28 11:08:32.583087 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:32.582925    2552 generic.go:358] "Generic (PLEG): container finished" podID="41941f76-780c-485d-8a77-140318ee6e89" containerID="e9746a9f248754d3355ad074542f11d4b5984c3b3de06a52ad3b133a5e4b9848" exitCode=2
May 28 11:08:32.583087 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:32.583016    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bff887bb5-6hswq" event={"ID":"41941f76-780c-485d-8a77-140318ee6e89","Type":"ContainerDied","Data":"e9746a9f248754d3355ad074542f11d4b5984c3b3de06a52ad3b133a5e4b9848"}
May 28 11:08:33.899947 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:33.899914    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-bff887bb5-6hswq_41941f76-780c-485d-8a77-140318ee6e89/console/0.log"
May 28 11:08:33.900340 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:33.899988    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:08:34.016997 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.016894    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-console-config\") pod \"41941f76-780c-485d-8a77-140318ee6e89\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") "
May 28 11:08:34.016997 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.016934    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6782x\" (UniqueName: \"kubernetes.io/projected/41941f76-780c-485d-8a77-140318ee6e89-kube-api-access-6782x\") pod \"41941f76-780c-485d-8a77-140318ee6e89\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") "
May 28 11:08:34.017261 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.017062    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-trusted-ca-bundle\") pod \"41941f76-780c-485d-8a77-140318ee6e89\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") "
May 28 11:08:34.017261 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.017092    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41941f76-780c-485d-8a77-140318ee6e89-console-oauth-config\") pod \"41941f76-780c-485d-8a77-140318ee6e89\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") "
May 28 11:08:34.017261 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.017119    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-service-ca\") pod \"41941f76-780c-485d-8a77-140318ee6e89\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") "
May 28 11:08:34.017261 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.017159    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41941f76-780c-485d-8a77-140318ee6e89-console-serving-cert\") pod \"41941f76-780c-485d-8a77-140318ee6e89\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") "
May 28 11:08:34.017261 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.017198    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-oauth-serving-cert\") pod \"41941f76-780c-485d-8a77-140318ee6e89\" (UID: \"41941f76-780c-485d-8a77-140318ee6e89\") "
May 28 11:08:34.017514 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.017313    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-console-config" (OuterVolumeSpecName: "console-config") pod "41941f76-780c-485d-8a77-140318ee6e89" (UID: "41941f76-780c-485d-8a77-140318ee6e89"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 11:08:34.017514 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.017443    2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-console-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:08:34.017666 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.017643    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "41941f76-780c-485d-8a77-140318ee6e89" (UID: "41941f76-780c-485d-8a77-140318ee6e89"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 11:08:34.017725 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.017636    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-service-ca" (OuterVolumeSpecName: "service-ca") pod "41941f76-780c-485d-8a77-140318ee6e89" (UID: "41941f76-780c-485d-8a77-140318ee6e89"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 11:08:34.017725 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.017680    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "41941f76-780c-485d-8a77-140318ee6e89" (UID: "41941f76-780c-485d-8a77-140318ee6e89"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 11:08:34.019377 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.019340    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41941f76-780c-485d-8a77-140318ee6e89-kube-api-access-6782x" (OuterVolumeSpecName: "kube-api-access-6782x") pod "41941f76-780c-485d-8a77-140318ee6e89" (UID: "41941f76-780c-485d-8a77-140318ee6e89"). InnerVolumeSpecName "kube-api-access-6782x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:08:34.019511 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.019412    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41941f76-780c-485d-8a77-140318ee6e89-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "41941f76-780c-485d-8a77-140318ee6e89" (UID: "41941f76-780c-485d-8a77-140318ee6e89"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:08:34.019572 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.019503    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41941f76-780c-485d-8a77-140318ee6e89-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "41941f76-780c-485d-8a77-140318ee6e89" (UID: "41941f76-780c-485d-8a77-140318ee6e89"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:08:34.117847 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.117803    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6782x\" (UniqueName: \"kubernetes.io/projected/41941f76-780c-485d-8a77-140318ee6e89-kube-api-access-6782x\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:08:34.117847 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.117839    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-trusted-ca-bundle\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:08:34.117847 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.117852    2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41941f76-780c-485d-8a77-140318ee6e89-console-oauth-config\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:08:34.118092 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.117865    2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-service-ca\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:08:34.118092 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.117879    2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41941f76-780c-485d-8a77-140318ee6e89-console-serving-cert\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:08:34.118092 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.117891    2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41941f76-780c-485d-8a77-140318ee6e89-oauth-serving-cert\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:08:34.590307 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.590277    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-bff887bb5-6hswq_41941f76-780c-485d-8a77-140318ee6e89/console/0.log"
May 28 11:08:34.590474 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.590386    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bff887bb5-6hswq"
May 28 11:08:34.590474 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.590385    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bff887bb5-6hswq" event={"ID":"41941f76-780c-485d-8a77-140318ee6e89","Type":"ContainerDied","Data":"4fa1ccc67d6b5add9223cd3107d96010b30928e1616caf47ae1702655653e6b4"}
May 28 11:08:34.590556 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.590505    2552 scope.go:117] "RemoveContainer" containerID="e9746a9f248754d3355ad074542f11d4b5984c3b3de06a52ad3b133a5e4b9848"
May 28 11:08:34.591915 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.591873    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4znlc" event={"ID":"4e5305c2-8214-4b76-9e9b-56fd284dbe37","Type":"ContainerStarted","Data":"2236356d461be428ffbaf7a0c10c1a7651bca7242a3cdf9551c43d8ea47df3f2"}
May 28 11:08:34.612774 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.612720    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4znlc" podStartSLOduration=2.237158861 podStartE2EDuration="6.612705187s" podCreationTimestamp="2026-05-28 11:08:28 +0000 UTC" firstStartedPulling="2026-05-28 11:08:29.439482348 +0000 UTC m=+325.478579304" lastFinishedPulling="2026-05-28 11:08:33.815028666 +0000 UTC m=+329.854125630" observedRunningTime="2026-05-28 11:08:34.610546588 +0000 UTC m=+330.649643563" watchObservedRunningTime="2026-05-28 11:08:34.612705187 +0000 UTC m=+330.651802166"
May 28 11:08:34.624990 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.624963    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bff887bb5-6hswq"]
May 28 11:08:34.628467 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:34.628438    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bff887bb5-6hswq"]
May 28 11:08:36.517658 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:08:36.517611    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41941f76-780c-485d-8a77-140318ee6e89" path="/var/lib/kubelet/pods/41941f76-780c-485d-8a77-140318ee6e89/volumes"
May 28 11:10:53.703269 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.703231    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"]
May 28 11:10:53.703841 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.703660    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41941f76-780c-485d-8a77-140318ee6e89" containerName="console"
May 28 11:10:53.703841 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.703673    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="41941f76-780c-485d-8a77-140318ee6e89" containerName="console"
May 28 11:10:53.703841 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.703753    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="41941f76-780c-485d-8a77-140318ee6e89" containerName="console"
May 28 11:10:53.706554 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.706535    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:53.709045 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.709021    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 28 11:10:53.709045 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.709035    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 11:10:53.709252 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.709021    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 28 11:10:53.709338 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.709320    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 28 11:10:53.709589 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.709573    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 28 11:10:53.709674 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.709629    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-qkg5s\""
May 28 11:10:53.709860 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.709846    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 11:10:53.709967 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.709953    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 28 11:10:53.720077 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.720056    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"]
May 28 11:10:53.770388 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.770367    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-ui-tls\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:53.770505 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.770398    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-registry-tls\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:53.770505 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.770425    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-offline-tls\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:53.770505 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.770452    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mvk2\" (UniqueName: \"kubernetes.io/projected/47219b60-2edf-4271-a603-1cd65b3ebfab-kube-api-access-7mvk2\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:53.770505 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.770471    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-online-tls\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:53.770705 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.770559    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/47219b60-2edf-4271-a603-1cd65b3ebfab-feast-data\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:53.870995 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.870954    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7mvk2\" (UniqueName: \"kubernetes.io/projected/47219b60-2edf-4271-a603-1cd65b3ebfab-kube-api-access-7mvk2\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:53.870995 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.870989    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-online-tls\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:53.871247 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.871037    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/47219b60-2edf-4271-a603-1cd65b3ebfab-feast-data\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:53.871247 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.871073    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-ui-tls\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:53.871247 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.871100    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-registry-tls\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:53.871247 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.871143    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-offline-tls\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:53.871247 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:10:53.871181    2552 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 28 11:10:53.871499 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:10:53.871254    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-online-tls podName:47219b60-2edf-4271-a603-1cd65b3ebfab nodeName:}" failed. No retries permitted until 2026-05-28 11:10:54.371232269 +0000 UTC m=+470.410329227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-online-tls") pod "feast-simple-feast-setup-8496444f68-hpcq6" (UID: "47219b60-2edf-4271-a603-1cd65b3ebfab") : secret "feast-simple-feast-setup-online-tls" not found
May 28 11:10:53.871499 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:10:53.871377    2552 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 28 11:10:53.871499 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:10:53.871432    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-ui-tls podName:47219b60-2edf-4271-a603-1cd65b3ebfab nodeName:}" failed. No retries permitted until 2026-05-28 11:10:54.371412671 +0000 UTC m=+470.410509634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-ui-tls") pod "feast-simple-feast-setup-8496444f68-hpcq6" (UID: "47219b60-2edf-4271-a603-1cd65b3ebfab") : secret "feast-simple-feast-setup-ui-tls" not found
May 28 11:10:53.871676 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:10:53.871512    2552 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 28 11:10:53.871676 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:10:53.871555    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-registry-tls podName:47219b60-2edf-4271-a603-1cd65b3ebfab nodeName:}" failed. No retries permitted until 2026-05-28 11:10:54.371542122 +0000 UTC m=+470.410639080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-registry-tls") pod "feast-simple-feast-setup-8496444f68-hpcq6" (UID: "47219b60-2edf-4271-a603-1cd65b3ebfab") : secret "feast-simple-feast-setup-registry-tls" not found
May 28 11:10:53.871917 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.871890    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/47219b60-2edf-4271-a603-1cd65b3ebfab-feast-data\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:53.873596 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.873575    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-offline-tls\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:53.879236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:53.879217    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mvk2\" (UniqueName: \"kubernetes.io/projected/47219b60-2edf-4271-a603-1cd65b3ebfab-kube-api-access-7mvk2\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:54.374836 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:54.374780    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-online-tls\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:54.375009 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:54.374895    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-ui-tls\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:54.375009 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:54.374921    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-registry-tls\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:54.377281 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:54.377251    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-ui-tls\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:54.377394 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:54.377306    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-online-tls\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:54.377394 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:54.377306    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-registry-tls\") pod \"feast-simple-feast-setup-8496444f68-hpcq6\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:54.619109 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:54.619077    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:10:54.745852 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:54.745822    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"]
May 28 11:10:54.749673 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:10:54.749638    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47219b60_2edf_4271_a603_1cd65b3ebfab.slice/crio-f6c926c9b6c9e72497c7653b94f093a59ee2febd735a2a8d295ed3de53f08730 WatchSource:0}: Error finding container f6c926c9b6c9e72497c7653b94f093a59ee2febd735a2a8d295ed3de53f08730: Status 404 returned error can't find the container with id f6c926c9b6c9e72497c7653b94f093a59ee2febd735a2a8d295ed3de53f08730
May 28 11:10:54.988986 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:10:54.988957    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" event={"ID":"47219b60-2edf-4271-a603-1cd65b3ebfab","Type":"ContainerStarted","Data":"f6c926c9b6c9e72497c7653b94f093a59ee2febd735a2a8d295ed3de53f08730"}
May 28 11:11:27.091643 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:27.091596    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" event={"ID":"47219b60-2edf-4271-a603-1cd65b3ebfab","Type":"ContainerStarted","Data":"bf9f2b0f0abc896fd9492628ac3b3bdb1a49083ac4b4d19ca53af63cffb5114f"}
May 28 11:11:31.103667 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:31.103633    2552 generic.go:358] "Generic (PLEG): container finished" podID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerID="bf9f2b0f0abc896fd9492628ac3b3bdb1a49083ac4b4d19ca53af63cffb5114f" exitCode=0
May 28 11:11:31.104022 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:31.103711    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" event={"ID":"47219b60-2edf-4271-a603-1cd65b3ebfab","Type":"ContainerDied","Data":"bf9f2b0f0abc896fd9492628ac3b3bdb1a49083ac4b4d19ca53af63cffb5114f"}
May 28 11:11:32.108355 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:32.108318    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" event={"ID":"47219b60-2edf-4271-a603-1cd65b3ebfab","Type":"ContainerStarted","Data":"575d9bbb551dbc9520ce33e92e104aa4b5ba27a3dfc74662d852f75a8279c5cb"}
May 28 11:11:37.123584 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:37.123548    2552 generic.go:358] "Generic (PLEG): container finished" podID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerID="575d9bbb551dbc9520ce33e92e104aa4b5ba27a3dfc74662d852f75a8279c5cb" exitCode=0
May 28 11:11:37.123958 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:37.123623    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" event={"ID":"47219b60-2edf-4271-a603-1cd65b3ebfab","Type":"ContainerDied","Data":"575d9bbb551dbc9520ce33e92e104aa4b5ba27a3dfc74662d852f75a8279c5cb"}
May 28 11:11:37.127784 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:37.127759    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 28 11:11:38.131463 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:38.131412    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" event={"ID":"47219b60-2edf-4271-a603-1cd65b3ebfab","Type":"ContainerStarted","Data":"91c80b5d3e1ec76cfb7987aa054cb09493e79b994270253486a4dc5c2b7b2a5f"}
May 28 11:11:38.131463 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:38.131463    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" event={"ID":"47219b60-2edf-4271-a603-1cd65b3ebfab","Type":"ContainerStarted","Data":"b6ba626586a964f5d67e68af243a793819292dccd0cda73e58797d8871912630"}
May 28 11:11:38.132004 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:38.131478    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" event={"ID":"47219b60-2edf-4271-a603-1cd65b3ebfab","Type":"ContainerStarted","Data":"533beba91867c54f08606d48f2ea7d273821f83a50d29db00d1ffc19ebe03f1e"}
May 28 11:11:38.132004 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:38.131491    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" event={"ID":"47219b60-2edf-4271-a603-1cd65b3ebfab","Type":"ContainerStarted","Data":"da969c686814ff773a7077ae6f6679d0cc6a555b9d0005466ce277d9555afc69"}
May 28 11:11:38.156366 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:38.156287    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podStartSLOduration=13.696985533 podStartE2EDuration="45.156266877s" podCreationTimestamp="2026-05-28 11:10:53 +0000 UTC" firstStartedPulling="2026-05-28 11:10:54.751457434 +0000 UTC m=+470.790554390" lastFinishedPulling="2026-05-28 11:11:26.210738775 +0000 UTC m=+502.249835734" observedRunningTime="2026-05-28 11:11:38.154592096 +0000 UTC m=+514.193689142" watchObservedRunningTime="2026-05-28 11:11:38.156266877 +0000 UTC m=+514.195363857"
May 28 11:11:39.620337 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:39.620037    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:11:39.620337 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:39.620089    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:11:39.620337 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:39.620103    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:11:39.620936 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:39.620665    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:11:39.622163 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:39.622095    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused"
May 28 11:11:39.622291 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:39.622157    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused"
May 28 11:11:39.622291 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:39.622184    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused"
May 28 11:11:39.622291 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:39.622268    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused"
May 28 11:11:42.620427 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:42.620382    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused"
May 28 11:11:42.621302 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:42.621280    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:11:42.621430 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:42.621405    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:11:42.621833 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:42.621809    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:11:42.621921 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:42.621844    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:11:42.622186 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:42.622169    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:11:42.622277 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:42.622259    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:11:42.652252 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:42.652226    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:11:43.150829 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:43.150792    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:11:43.153684 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:43.153663    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:11:44.619551 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:44.619504    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:11:45.620825 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:45.620795    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:11:46.161348 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:11:46.161318    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:12:48.938423 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:48.938390    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"]
May 28 11:12:48.939076 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:48.938726    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="registry" containerID="cri-o://da969c686814ff773a7077ae6f6679d0cc6a555b9d0005466ce277d9555afc69" gracePeriod=30
May 28 11:12:48.939076 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:48.938893    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="ui" containerID="cri-o://91c80b5d3e1ec76cfb7987aa054cb09493e79b994270253486a4dc5c2b7b2a5f" gracePeriod=30
May 28 11:12:48.939076 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:48.938947    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="online" containerID="cri-o://533beba91867c54f08606d48f2ea7d273821f83a50d29db00d1ffc19ebe03f1e" gracePeriod=30
May 28 11:12:48.939076 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:48.939010    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="offline" containerID="cri-o://b6ba626586a964f5d67e68af243a793819292dccd0cda73e58797d8871912630" gracePeriod=30
May 28 11:12:49.192875 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.192794    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"]
May 28 11:12:49.196586 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.196555    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.199535 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.199514    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-fk2w6\""
May 28 11:12:49.209545 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.209523    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"]
May 28 11:12:49.272100 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.272052    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-offline-tls\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.272290 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.272243    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-registry-tls\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.272341 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.272302    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m9h2\" (UniqueName: \"kubernetes.io/projected/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-kube-api-access-8m9h2\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.272425 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.272405    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-online-tls\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.272481 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.272461    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-ui-tls\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.272551 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.272532    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-feast-data\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.373501 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.373462    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-online-tls\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.373688 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.373518    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-ui-tls\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.373688 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.373555    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-feast-data\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.373688 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.373604    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-offline-tls\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.373688 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.373654    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-registry-tls\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.373688 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.373680    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8m9h2\" (UniqueName: \"kubernetes.io/projected/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-kube-api-access-8m9h2\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.374065 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:12:49.374038    2552 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 28 11:12:49.374228 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:12:49.374111    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-registry-tls podName:bc7dd935-673c-4e4a-ab3a-bfcd427170ee nodeName:}" failed. No retries permitted until 2026-05-28 11:12:49.874089756 +0000 UTC m=+585.913186714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-registry-tls") pod "feast-simple-feast-setup-8496444f68-6dnp9" (UID: "bc7dd935-673c-4e4a-ab3a-bfcd427170ee") : secret "feast-simple-feast-setup-registry-tls" not found
May 28 11:12:49.374389 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.374366    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-feast-data\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.376463 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.376438    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-offline-tls\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.376587 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.376441    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-ui-tls\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.376772 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.376753    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-online-tls\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.382438 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.382408    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m9h2\" (UniqueName: \"kubernetes.io/projected/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-kube-api-access-8m9h2\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.879367 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.879340    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-registry-tls\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:49.881600 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:49.881583    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-registry-tls\") pod \"feast-simple-feast-setup-8496444f68-6dnp9\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") " pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:50.114080 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:50.114055    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:12:50.242073 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:50.242044    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"]
May 28 11:12:50.244064 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:12:50.244032    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc7dd935_673c_4e4a_ab3a_bfcd427170ee.slice/crio-387faf4562e2257d2e499253d2c412a15c004d06f55c92e24cd1c3aedf1c1b8e WatchSource:0}: Error finding container 387faf4562e2257d2e499253d2c412a15c004d06f55c92e24cd1c3aedf1c1b8e: Status 404 returned error can't find the container with id 387faf4562e2257d2e499253d2c412a15c004d06f55c92e24cd1c3aedf1c1b8e
May 28 11:12:50.346335 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:50.346306    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" event={"ID":"bc7dd935-673c-4e4a-ab3a-bfcd427170ee","Type":"ContainerStarted","Data":"cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1"}
May 28 11:12:50.346477 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:50.346344    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" event={"ID":"bc7dd935-673c-4e4a-ab3a-bfcd427170ee","Type":"ContainerStarted","Data":"387faf4562e2257d2e499253d2c412a15c004d06f55c92e24cd1c3aedf1c1b8e"}
May 28 11:12:50.349004 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:50.348977    2552 generic.go:358] "Generic (PLEG): container finished" podID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerID="91c80b5d3e1ec76cfb7987aa054cb09493e79b994270253486a4dc5c2b7b2a5f" exitCode=0
May 28 11:12:50.349004 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:50.349003    2552 generic.go:358] "Generic (PLEG): container finished" podID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerID="533beba91867c54f08606d48f2ea7d273821f83a50d29db00d1ffc19ebe03f1e" exitCode=0
May 28 11:12:50.349157 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:50.349034    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" event={"ID":"47219b60-2edf-4271-a603-1cd65b3ebfab","Type":"ContainerDied","Data":"91c80b5d3e1ec76cfb7987aa054cb09493e79b994270253486a4dc5c2b7b2a5f"}
May 28 11:12:50.349157 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:50.349052    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" event={"ID":"47219b60-2edf-4271-a603-1cd65b3ebfab","Type":"ContainerDied","Data":"533beba91867c54f08606d48f2ea7d273821f83a50d29db00d1ffc19ebe03f1e"}
May 28 11:12:53.151498 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:53.151448    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused"
May 28 11:12:55.366789 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:55.366755    2552 generic.go:358] "Generic (PLEG): container finished" podID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerID="cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1" exitCode=0
May 28 11:12:55.367202 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:55.366836    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" event={"ID":"bc7dd935-673c-4e4a-ab3a-bfcd427170ee","Type":"ContainerDied","Data":"cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1"}
May 28 11:12:56.161522 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:56.161481    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused"
May 28 11:12:56.371693 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:12:56.371650    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" event={"ID":"bc7dd935-673c-4e4a-ab3a-bfcd427170ee","Type":"ContainerStarted","Data":"bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205"}
May 28 11:13:01.387706 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:01.387672    2552 generic.go:358] "Generic (PLEG): container finished" podID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerID="bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205" exitCode=0
May 28 11:13:01.388082 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:01.387727    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" event={"ID":"bc7dd935-673c-4e4a-ab3a-bfcd427170ee","Type":"ContainerDied","Data":"bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205"}
May 28 11:13:02.395641 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:02.395597    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" event={"ID":"bc7dd935-673c-4e4a-ab3a-bfcd427170ee","Type":"ContainerStarted","Data":"6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8"}
May 28 11:13:02.396150 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:02.395650    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" event={"ID":"bc7dd935-673c-4e4a-ab3a-bfcd427170ee","Type":"ContainerStarted","Data":"ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc"}
May 28 11:13:02.396150 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:02.395665    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" event={"ID":"bc7dd935-673c-4e4a-ab3a-bfcd427170ee","Type":"ContainerStarted","Data":"8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f"}
May 28 11:13:02.396150 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:02.395677    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" event={"ID":"bc7dd935-673c-4e4a-ab3a-bfcd427170ee","Type":"ContainerStarted","Data":"4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf"}
May 28 11:13:02.421579 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:02.421515    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" podStartSLOduration=13.421497112 podStartE2EDuration="13.421497112s" podCreationTimestamp="2026-05-28 11:12:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 11:13:02.420721355 +0000 UTC m=+598.459818335" watchObservedRunningTime="2026-05-28 11:13:02.421497112 +0000 UTC m=+598.460594119"
May 28 11:13:03.151112 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:03.151060    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused"
May 28 11:13:04.433085 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:04.433042    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:13:04.433674 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:04.433087    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:13:05.114479 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:05.114400    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:05.114479 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:05.114455    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:05.114479 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:05.114472    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:05.114777 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:05.114584    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:05.117080 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:05.117046    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused"
May 28 11:13:05.117337 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:05.117075    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused"
May 28 11:13:05.117542 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:05.117103    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 28 11:13:05.117646 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:05.117115    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused"
May 28 11:13:06.160981 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:06.160935    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused"
May 28 11:13:08.115977 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:08.115944    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:08.117427 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:08.116124    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:08.117427 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:08.116271    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:08.117427 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:08.116585    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:08.117427 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:08.116612    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:08.117427 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:08.116625    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:08.117427 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:08.116793    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:08.117427 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:08.116834    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:08.117427 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:08.117008    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:08.120585 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:08.120570    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:08.416356 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:08.416277    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:08.419197 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:08.419172    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:13:13.151476 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:13.151437    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused"
May 28 11:13:13.151932 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:13.151550    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:13:16.161288 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:16.161244    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused"
May 28 11:13:16.161661 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:16.161378    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:13:19.448729 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.448702    2552 generic.go:358] "Generic (PLEG): container finished" podID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerID="b6ba626586a964f5d67e68af243a793819292dccd0cda73e58797d8871912630" exitCode=137
May 28 11:13:19.448729 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.448728    2552 generic.go:358] "Generic (PLEG): container finished" podID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerID="da969c686814ff773a7077ae6f6679d0cc6a555b9d0005466ce277d9555afc69" exitCode=137
May 28 11:13:19.449155 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.448782    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" event={"ID":"47219b60-2edf-4271-a603-1cd65b3ebfab","Type":"ContainerDied","Data":"b6ba626586a964f5d67e68af243a793819292dccd0cda73e58797d8871912630"}
May 28 11:13:19.449155 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.448823    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" event={"ID":"47219b60-2edf-4271-a603-1cd65b3ebfab","Type":"ContainerDied","Data":"da969c686814ff773a7077ae6f6679d0cc6a555b9d0005466ce277d9555afc69"}
May 28 11:13:19.583749 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.583728    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:13:19.732094 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.732071    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-registry-tls\") pod \"47219b60-2edf-4271-a603-1cd65b3ebfab\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") "
May 28 11:13:19.732248 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.732153    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-ui-tls\") pod \"47219b60-2edf-4271-a603-1cd65b3ebfab\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") "
May 28 11:13:19.732248 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.732181    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-offline-tls\") pod \"47219b60-2edf-4271-a603-1cd65b3ebfab\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") "
May 28 11:13:19.732248 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.732214    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mvk2\" (UniqueName: \"kubernetes.io/projected/47219b60-2edf-4271-a603-1cd65b3ebfab-kube-api-access-7mvk2\") pod \"47219b60-2edf-4271-a603-1cd65b3ebfab\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") "
May 28 11:13:19.732419 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.732264    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-online-tls\") pod \"47219b60-2edf-4271-a603-1cd65b3ebfab\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") "
May 28 11:13:19.732419 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.732290    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/47219b60-2edf-4271-a603-1cd65b3ebfab-feast-data\") pod \"47219b60-2edf-4271-a603-1cd65b3ebfab\" (UID: \"47219b60-2edf-4271-a603-1cd65b3ebfab\") "
May 28 11:13:19.732928 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.732898    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47219b60-2edf-4271-a603-1cd65b3ebfab-feast-data" (OuterVolumeSpecName: "feast-data") pod "47219b60-2edf-4271-a603-1cd65b3ebfab" (UID: "47219b60-2edf-4271-a603-1cd65b3ebfab"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 11:13:19.734448 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.734415    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "47219b60-2edf-4271-a603-1cd65b3ebfab" (UID: "47219b60-2edf-4271-a603-1cd65b3ebfab"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:13:19.734541 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.734467    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "47219b60-2edf-4271-a603-1cd65b3ebfab" (UID: "47219b60-2edf-4271-a603-1cd65b3ebfab"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:13:19.734541 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.734492    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47219b60-2edf-4271-a603-1cd65b3ebfab-kube-api-access-7mvk2" (OuterVolumeSpecName: "kube-api-access-7mvk2") pod "47219b60-2edf-4271-a603-1cd65b3ebfab" (UID: "47219b60-2edf-4271-a603-1cd65b3ebfab"). InnerVolumeSpecName "kube-api-access-7mvk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:13:19.734686 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.734655    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-online-tls" (OuterVolumeSpecName: "online-tls") pod "47219b60-2edf-4271-a603-1cd65b3ebfab" (UID: "47219b60-2edf-4271-a603-1cd65b3ebfab"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:13:19.734686 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.734678    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "47219b60-2edf-4271-a603-1cd65b3ebfab" (UID: "47219b60-2edf-4271-a603-1cd65b3ebfab"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:13:19.833574 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.833541    2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-registry-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:13:19.833574 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.833572    2552 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-ui-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:13:19.833574 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.833581    2552 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-offline-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:13:19.833736 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.833590    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mvk2\" (UniqueName: \"kubernetes.io/projected/47219b60-2edf-4271-a603-1cd65b3ebfab-kube-api-access-7mvk2\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:13:19.833736 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.833600    2552 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47219b60-2edf-4271-a603-1cd65b3ebfab-online-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:13:19.833736 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:19.833610    2552 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/47219b60-2edf-4271-a603-1cd65b3ebfab-feast-data\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:13:20.454002 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:20.453976    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"
May 28 11:13:20.454417 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:20.453978    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6" event={"ID":"47219b60-2edf-4271-a603-1cd65b3ebfab","Type":"ContainerDied","Data":"f6c926c9b6c9e72497c7653b94f093a59ee2febd735a2a8d295ed3de53f08730"}
May 28 11:13:20.454417 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:20.454095    2552 scope.go:117] "RemoveContainer" containerID="91c80b5d3e1ec76cfb7987aa054cb09493e79b994270253486a4dc5c2b7b2a5f"
May 28 11:13:20.463612 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:20.463594    2552 scope.go:117] "RemoveContainer" containerID="b6ba626586a964f5d67e68af243a793819292dccd0cda73e58797d8871912630"
May 28 11:13:20.470685 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:20.470669    2552 scope.go:117] "RemoveContainer" containerID="533beba91867c54f08606d48f2ea7d273821f83a50d29db00d1ffc19ebe03f1e"
May 28 11:13:20.475033 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:20.475013    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"]
May 28 11:13:20.478218 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:20.478175    2552 scope.go:117] "RemoveContainer" containerID="da969c686814ff773a7077ae6f6679d0cc6a555b9d0005466ce277d9555afc69"
May 28 11:13:20.478932 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:20.478905    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8496444f68-hpcq6"]
May 28 11:13:20.485043 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:20.485027    2552 scope.go:117] "RemoveContainer" containerID="575d9bbb551dbc9520ce33e92e104aa4b5ba27a3dfc74662d852f75a8279c5cb"
May 28 11:13:20.491434 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:20.491415    2552 scope.go:117] "RemoveContainer" containerID="bf9f2b0f0abc896fd9492628ac3b3bdb1a49083ac4b4d19ca53af63cffb5114f"
May 28 11:13:20.516619 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:13:20.516592    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" path="/var/lib/kubelet/pods/47219b60-2edf-4271-a603-1cd65b3ebfab/volumes"
May 28 11:14:10.800704 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.800668    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"]
May 28 11:14:10.801121 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801001    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="online"
May 28 11:14:10.801121 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801013    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="online"
May 28 11:14:10.801121 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801026    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="feast-apply"
May 28 11:14:10.801121 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801032    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="feast-apply"
May 28 11:14:10.801121 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801038    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="registry"
May 28 11:14:10.801121 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801044    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="registry"
May 28 11:14:10.801121 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801057    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="ui"
May 28 11:14:10.801121 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801065    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="ui"
May 28 11:14:10.801121 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801085    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="offline"
May 28 11:14:10.801121 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801090    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="offline"
May 28 11:14:10.801121 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801097    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="feast-init"
May 28 11:14:10.801121 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801102    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="feast-init"
May 28 11:14:10.801500 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801177    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="online"
May 28 11:14:10.801500 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801187    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="offline"
May 28 11:14:10.801500 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801194    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="registry"
May 28 11:14:10.801500 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.801200    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="47219b60-2edf-4271-a603-1cd65b3ebfab" containerName="ui"
May 28 11:14:10.803922 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.803904    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:10.806034 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.806009    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 28 11:14:10.806275 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.806246    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 28 11:14:10.806380 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.806278    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 28 11:14:10.806380 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.806369    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-bt5vg\""
May 28 11:14:10.806505 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.806487    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 28 11:14:10.806637 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.806617    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 28 11:14:10.806719 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.806592    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\""
May 28 11:14:10.806777 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.806567    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 28 11:14:10.815905 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.815871    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"]
May 28 11:14:10.921822 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.921786    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2tjm\" (UniqueName: \"kubernetes.io/projected/0b79b6bb-bd7e-443d-ad48-35d112237c97-kube-api-access-g2tjm\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:10.921822 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.921832    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0b79b6bb-bd7e-443d-ad48-35d112237c97-feast-data\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:10.922091 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.921857    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-online-tls\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:10.922091 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.921945    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0b79b6bb-bd7e-443d-ad48-35d112237c97-registry-tls\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:10.922091 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.921987    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-offline-tls\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:10.922091 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:10.922056    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-ui-tls\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.023083 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.023039    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g2tjm\" (UniqueName: \"kubernetes.io/projected/0b79b6bb-bd7e-443d-ad48-35d112237c97-kube-api-access-g2tjm\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.023312 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.023102    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0b79b6bb-bd7e-443d-ad48-35d112237c97-feast-data\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.023312 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.023152    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-online-tls\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.023312 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.023186    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0b79b6bb-bd7e-443d-ad48-35d112237c97-registry-tls\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.023312 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.023215    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-offline-tls\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.023312 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.023252    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-ui-tls\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.023312 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:14:11.023263    2552 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found
May 28 11:14:11.023626 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:14:11.023345    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-online-tls podName:0b79b6bb-bd7e-443d-ad48-35d112237c97 nodeName:}" failed. No retries permitted until 2026-05-28 11:14:11.523324533 +0000 UTC m=+667.562421508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-online-tls") pod "feast-simple-feast-remote-setup-559c7f679b-t96zt" (UID: "0b79b6bb-bd7e-443d-ad48-35d112237c97") : secret "feast-simple-feast-remote-setup-online-tls" not found
May 28 11:14:11.023626 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:14:11.023383    2552 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found
May 28 11:14:11.023626 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:14:11.023444    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-ui-tls podName:0b79b6bb-bd7e-443d-ad48-35d112237c97 nodeName:}" failed. No retries permitted until 2026-05-28 11:14:11.523426375 +0000 UTC m=+667.562523336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-ui-tls") pod "feast-simple-feast-remote-setup-559c7f679b-t96zt" (UID: "0b79b6bb-bd7e-443d-ad48-35d112237c97") : secret "feast-simple-feast-remote-setup-ui-tls" not found
May 28 11:14:11.023626 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.023573    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0b79b6bb-bd7e-443d-ad48-35d112237c97-feast-data\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.024387 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.024359    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0b79b6bb-bd7e-443d-ad48-35d112237c97-registry-tls\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.025796 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.025773    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-offline-tls\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.031050 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.031027    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2tjm\" (UniqueName: \"kubernetes.io/projected/0b79b6bb-bd7e-443d-ad48-35d112237c97-kube-api-access-g2tjm\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.528106 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.528061    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-online-tls\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.528332 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.528121    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-ui-tls\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.530520 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.530491    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-ui-tls\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.530617 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.530551    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-online-tls\") pod \"feast-simple-feast-remote-setup-559c7f679b-t96zt\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.714617 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.714583    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:11.844825 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.844791    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"]
May 28 11:14:11.848002 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:14:11.847959    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b79b6bb_bd7e_443d_ad48_35d112237c97.slice/crio-9fa6beb2f111cb40fb11e14328d075736781095357041ef43830de6f8e15e137 WatchSource:0}: Error finding container 9fa6beb2f111cb40fb11e14328d075736781095357041ef43830de6f8e15e137: Status 404 returned error can't find the container with id 9fa6beb2f111cb40fb11e14328d075736781095357041ef43830de6f8e15e137
May 28 11:14:11.849733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:11.849718    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 11:14:12.609068 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:12.609026    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" event={"ID":"0b79b6bb-bd7e-443d-ad48-35d112237c97","Type":"ContainerStarted","Data":"acc7934fec72eba092e62a43a4fcfb24ae9109422e2b17e23cb1b2388c3e891f"}
May 28 11:14:12.609068 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:12.609073    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" event={"ID":"0b79b6bb-bd7e-443d-ad48-35d112237c97","Type":"ContainerStarted","Data":"9fa6beb2f111cb40fb11e14328d075736781095357041ef43830de6f8e15e137"}
May 28 11:14:16.628472 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:16.628387    2552 generic.go:358] "Generic (PLEG): container finished" podID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerID="acc7934fec72eba092e62a43a4fcfb24ae9109422e2b17e23cb1b2388c3e891f" exitCode=0
May 28 11:14:16.628472 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:16.628450    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" event={"ID":"0b79b6bb-bd7e-443d-ad48-35d112237c97","Type":"ContainerDied","Data":"acc7934fec72eba092e62a43a4fcfb24ae9109422e2b17e23cb1b2388c3e891f"}
May 28 11:14:17.633552 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:17.633518    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" event={"ID":"0b79b6bb-bd7e-443d-ad48-35d112237c97","Type":"ContainerStarted","Data":"4ba3f44852b6594782927b656110929953523ed2edcd667192e76740dc622790"}
May 28 11:14:22.648972 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:22.648942    2552 generic.go:358] "Generic (PLEG): container finished" podID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerID="4ba3f44852b6594782927b656110929953523ed2edcd667192e76740dc622790" exitCode=0
May 28 11:14:22.649399 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:22.649017    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" event={"ID":"0b79b6bb-bd7e-443d-ad48-35d112237c97","Type":"ContainerDied","Data":"4ba3f44852b6594782927b656110929953523ed2edcd667192e76740dc622790"}
May 28 11:14:23.655148 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:23.655096    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" event={"ID":"0b79b6bb-bd7e-443d-ad48-35d112237c97","Type":"ContainerStarted","Data":"bce671267b120ed4ab5b16484325cec7aa8c092a1edfb9edf4abfd26919e6f6e"}
May 28 11:14:23.655494 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:23.655176    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" event={"ID":"0b79b6bb-bd7e-443d-ad48-35d112237c97","Type":"ContainerStarted","Data":"f804c7b1b511d6890d4f7f8b756eb1528cf4ec1dd79b6444a646e9fa89082754"}
May 28 11:14:23.655494 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:23.655195    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" event={"ID":"0b79b6bb-bd7e-443d-ad48-35d112237c97","Type":"ContainerStarted","Data":"650e3b7caf5aa12d6c4c94e9fca21ffb148d0588050ec4278d2edcf2297675be"}
May 28 11:14:23.677685 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:23.677619    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podStartSLOduration=13.677600802 podStartE2EDuration="13.677600802s" podCreationTimestamp="2026-05-28 11:14:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 11:14:23.675244238 +0000 UTC m=+679.714341217" watchObservedRunningTime="2026-05-28 11:14:23.677600802 +0000 UTC m=+679.716697783"
May 28 11:14:23.715225 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:23.715123    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:23.715225 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:23.715195    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:23.715225 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:23.715210    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:23.717309 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:23.717268    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused"
May 28 11:14:23.717409 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:23.717318    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 28 11:14:23.717409 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:23.717301    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.27:8816: connect: connection refused"
May 28 11:14:26.715275 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:26.715230    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.27:8816: connect: connection refused"
May 28 11:14:26.715761 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:26.715271    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 28 11:14:26.715761 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:26.715229    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused"
May 28 11:14:29.716079 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:29.716046    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:29.716597 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:29.716252    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:29.716657 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:29.716626    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:29.716657 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:29.716651    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:29.716851 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:29.716832    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:29.717022 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:29.717005    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:29.720367 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:29.720348    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:30.677634 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:30.677608    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:14:30.685086 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:14:30.685061    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:15:32.087429 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:32.087384    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"]
May 28 11:15:32.088011 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:32.087878    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="online" containerID="cri-o://650e3b7caf5aa12d6c4c94e9fca21ffb148d0588050ec4278d2edcf2297675be" gracePeriod=30
May 28 11:15:32.088097 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:32.088066    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="ui" containerID="cri-o://bce671267b120ed4ab5b16484325cec7aa8c092a1edfb9edf4abfd26919e6f6e" gracePeriod=30
May 28 11:15:32.088296 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:32.088164    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="offline" containerID="cri-o://f804c7b1b511d6890d4f7f8b756eb1528cf4ec1dd79b6444a646e9fa89082754" gracePeriod=30
May 28 11:15:32.202937 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:32.202901    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"]
May 28 11:15:32.203402 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:32.203282    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="registry" containerID="cri-o://4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf" gracePeriod=30
May 28 11:15:32.203402 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:32.203332    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="offline" containerID="cri-o://ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc" gracePeriod=30
May 28 11:15:32.203598 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:32.203338    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="online" containerID="cri-o://8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f" gracePeriod=30
May 28 11:15:32.203598 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:32.203346    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="ui" containerID="cri-o://6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8" gracePeriod=30
May 28 11:15:32.858324 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:32.858293    2552 generic.go:358] "Generic (PLEG): container finished" podID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerID="bce671267b120ed4ab5b16484325cec7aa8c092a1edfb9edf4abfd26919e6f6e" exitCode=0
May 28 11:15:32.858324 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:32.858321    2552 generic.go:358] "Generic (PLEG): container finished" podID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerID="650e3b7caf5aa12d6c4c94e9fca21ffb148d0588050ec4278d2edcf2297675be" exitCode=0
May 28 11:15:32.858476 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:32.858366    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" event={"ID":"0b79b6bb-bd7e-443d-ad48-35d112237c97","Type":"ContainerDied","Data":"bce671267b120ed4ab5b16484325cec7aa8c092a1edfb9edf4abfd26919e6f6e"}
May 28 11:15:32.858476 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:32.858406    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" event={"ID":"0b79b6bb-bd7e-443d-ad48-35d112237c97","Type":"ContainerDied","Data":"650e3b7caf5aa12d6c4c94e9fca21ffb148d0588050ec4278d2edcf2297675be"}
May 28 11:15:32.863515 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:32.863495    2552 generic.go:358] "Generic (PLEG): container finished" podID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerID="6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8" exitCode=0
May 28 11:15:32.863611 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:32.863529    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" event={"ID":"bc7dd935-673c-4e4a-ab3a-bfcd427170ee","Type":"ContainerDied","Data":"6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8"}
May 28 11:15:33.871475 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:33.871443    2552 generic.go:358] "Generic (PLEG): container finished" podID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerID="8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f" exitCode=0
May 28 11:15:33.871821 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:33.871509    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" event={"ID":"bc7dd935-673c-4e4a-ab3a-bfcd427170ee","Type":"ContainerDied","Data":"8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f"}
May 28 11:15:38.117181 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:38.117099    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused"
May 28 11:15:38.416525 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:38.416430    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 28 11:15:39.717574 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:39.717527    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused"
May 28 11:15:40.678778 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:40.678738    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 28 11:15:48.117359 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:48.117316    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused"
May 28 11:15:48.416861 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:48.416773    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 28 11:15:49.716770 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:49.716729    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused"
May 28 11:15:50.678357 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:50.678318    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 28 11:15:58.117466 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:58.117419    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused"
May 28 11:15:58.117888 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:58.117574    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:15:58.416633 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:58.416546    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused"
May 28 11:15:58.416771 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:58.416655    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:15:59.716904 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:59.716861    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.27:8443: connect: connection refused"
May 28 11:15:59.717307 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:15:59.716997    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:16:00.678336 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:00.678301    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused"
May 28 11:16:00.678517 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:00.678424    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:16:02.733345 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.733322    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:16:02.849932 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.849910    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:16:02.893216 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.893150    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0b79b6bb-bd7e-443d-ad48-35d112237c97-feast-data\") pod \"0b79b6bb-bd7e-443d-ad48-35d112237c97\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") "
May 28 11:16:02.893216 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.893192    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-offline-tls\") pod \"0b79b6bb-bd7e-443d-ad48-35d112237c97\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") "
May 28 11:16:02.893410 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.893234    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0b79b6bb-bd7e-443d-ad48-35d112237c97-registry-tls\") pod \"0b79b6bb-bd7e-443d-ad48-35d112237c97\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") "
May 28 11:16:02.893410 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.893259    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-ui-tls\") pod \"0b79b6bb-bd7e-443d-ad48-35d112237c97\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") "
May 28 11:16:02.893410 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.893389    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-online-tls\") pod \"0b79b6bb-bd7e-443d-ad48-35d112237c97\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") "
May 28 11:16:02.893573 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.893475    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2tjm\" (UniqueName: \"kubernetes.io/projected/0b79b6bb-bd7e-443d-ad48-35d112237c97-kube-api-access-g2tjm\") pod \"0b79b6bb-bd7e-443d-ad48-35d112237c97\" (UID: \"0b79b6bb-bd7e-443d-ad48-35d112237c97\") "
May 28 11:16:02.893628 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.893591    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b79b6bb-bd7e-443d-ad48-35d112237c97-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0b79b6bb-bd7e-443d-ad48-35d112237c97" (UID: "0b79b6bb-bd7e-443d-ad48-35d112237c97"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 11:16:02.893684 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.893657    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b79b6bb-bd7e-443d-ad48-35d112237c97-feast-data" (OuterVolumeSpecName: "feast-data") pod "0b79b6bb-bd7e-443d-ad48-35d112237c97" (UID: "0b79b6bb-bd7e-443d-ad48-35d112237c97"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 11:16:02.893773 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.893754    2552 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0b79b6bb-bd7e-443d-ad48-35d112237c97-feast-data\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:16:02.893842 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.893781    2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0b79b6bb-bd7e-443d-ad48-35d112237c97-registry-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:16:02.895700 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.895669    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-online-tls" (OuterVolumeSpecName: "online-tls") pod "0b79b6bb-bd7e-443d-ad48-35d112237c97" (UID: "0b79b6bb-bd7e-443d-ad48-35d112237c97"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:16:02.895816 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.895799    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "0b79b6bb-bd7e-443d-ad48-35d112237c97" (UID: "0b79b6bb-bd7e-443d-ad48-35d112237c97"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:16:02.895878 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.895860    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b79b6bb-bd7e-443d-ad48-35d112237c97-kube-api-access-g2tjm" (OuterVolumeSpecName: "kube-api-access-g2tjm") pod "0b79b6bb-bd7e-443d-ad48-35d112237c97" (UID: "0b79b6bb-bd7e-443d-ad48-35d112237c97"). InnerVolumeSpecName "kube-api-access-g2tjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:16:02.895915 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.895884    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "0b79b6bb-bd7e-443d-ad48-35d112237c97" (UID: "0b79b6bb-bd7e-443d-ad48-35d112237c97"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:16:02.956114 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.956084    2552 generic.go:358] "Generic (PLEG): container finished" podID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerID="f804c7b1b511d6890d4f7f8b756eb1528cf4ec1dd79b6444a646e9fa89082754" exitCode=137
May 28 11:16:02.956299 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.956182    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"
May 28 11:16:02.956299 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.956178    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" event={"ID":"0b79b6bb-bd7e-443d-ad48-35d112237c97","Type":"ContainerDied","Data":"f804c7b1b511d6890d4f7f8b756eb1528cf4ec1dd79b6444a646e9fa89082754"}
May 28 11:16:02.956299 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.956224    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt" event={"ID":"0b79b6bb-bd7e-443d-ad48-35d112237c97","Type":"ContainerDied","Data":"9fa6beb2f111cb40fb11e14328d075736781095357041ef43830de6f8e15e137"}
May 28 11:16:02.956299 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.956245    2552 scope.go:117] "RemoveContainer" containerID="bce671267b120ed4ab5b16484325cec7aa8c092a1edfb9edf4abfd26919e6f6e"
May 28 11:16:02.958828 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.958801    2552 generic.go:358] "Generic (PLEG): container finished" podID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerID="ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc" exitCode=137
May 28 11:16:02.958828 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.958826    2552 generic.go:358] "Generic (PLEG): container finished" podID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerID="4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf" exitCode=137
May 28 11:16:02.958971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.958863    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" event={"ID":"bc7dd935-673c-4e4a-ab3a-bfcd427170ee","Type":"ContainerDied","Data":"ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc"}
May 28 11:16:02.958971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.958884    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" event={"ID":"bc7dd935-673c-4e4a-ab3a-bfcd427170ee","Type":"ContainerDied","Data":"4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf"}
May 28 11:16:02.958971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.958894    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9" event={"ID":"bc7dd935-673c-4e4a-ab3a-bfcd427170ee","Type":"ContainerDied","Data":"387faf4562e2257d2e499253d2c412a15c004d06f55c92e24cd1c3aedf1c1b8e"}
May 28 11:16:02.958971 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.958892    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"
May 28 11:16:02.965050 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.965020    2552 scope.go:117] "RemoveContainer" containerID="f804c7b1b511d6890d4f7f8b756eb1528cf4ec1dd79b6444a646e9fa89082754"
May 28 11:16:02.971972 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.971954    2552 scope.go:117] "RemoveContainer" containerID="650e3b7caf5aa12d6c4c94e9fca21ffb148d0588050ec4278d2edcf2297675be"
May 28 11:16:02.977633 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.977611    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"]
May 28 11:16:02.979534 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.979517    2552 scope.go:117] "RemoveContainer" containerID="4ba3f44852b6594782927b656110929953523ed2edcd667192e76740dc622790"
May 28 11:16:02.981749 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.981730    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-559c7f679b-t96zt"]
May 28 11:16:02.986619 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.986603    2552 scope.go:117] "RemoveContainer" containerID="acc7934fec72eba092e62a43a4fcfb24ae9109422e2b17e23cb1b2388c3e891f"
May 28 11:16:02.993072 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.993058    2552 scope.go:117] "RemoveContainer" containerID="bce671267b120ed4ab5b16484325cec7aa8c092a1edfb9edf4abfd26919e6f6e"
May 28 11:16:02.993319 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:16:02.993301    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce671267b120ed4ab5b16484325cec7aa8c092a1edfb9edf4abfd26919e6f6e\": container with ID starting with bce671267b120ed4ab5b16484325cec7aa8c092a1edfb9edf4abfd26919e6f6e not found: ID does not exist" containerID="bce671267b120ed4ab5b16484325cec7aa8c092a1edfb9edf4abfd26919e6f6e"
May 28 11:16:02.993382 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.993327    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce671267b120ed4ab5b16484325cec7aa8c092a1edfb9edf4abfd26919e6f6e"} err="failed to get container status \"bce671267b120ed4ab5b16484325cec7aa8c092a1edfb9edf4abfd26919e6f6e\": rpc error: code = NotFound desc = could not find container \"bce671267b120ed4ab5b16484325cec7aa8c092a1edfb9edf4abfd26919e6f6e\": container with ID starting with bce671267b120ed4ab5b16484325cec7aa8c092a1edfb9edf4abfd26919e6f6e not found: ID does not exist"
May 28 11:16:02.993382 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.993344    2552 scope.go:117] "RemoveContainer" containerID="f804c7b1b511d6890d4f7f8b756eb1528cf4ec1dd79b6444a646e9fa89082754"
May 28 11:16:02.993541 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:16:02.993527    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f804c7b1b511d6890d4f7f8b756eb1528cf4ec1dd79b6444a646e9fa89082754\": container with ID starting with f804c7b1b511d6890d4f7f8b756eb1528cf4ec1dd79b6444a646e9fa89082754 not found: ID does not exist" containerID="f804c7b1b511d6890d4f7f8b756eb1528cf4ec1dd79b6444a646e9fa89082754"
May 28 11:16:02.993579 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.993543    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f804c7b1b511d6890d4f7f8b756eb1528cf4ec1dd79b6444a646e9fa89082754"} err="failed to get container status \"f804c7b1b511d6890d4f7f8b756eb1528cf4ec1dd79b6444a646e9fa89082754\": rpc error: code = NotFound desc = could not find container \"f804c7b1b511d6890d4f7f8b756eb1528cf4ec1dd79b6444a646e9fa89082754\": container with ID starting with f804c7b1b511d6890d4f7f8b756eb1528cf4ec1dd79b6444a646e9fa89082754 not found: ID does not exist"
May 28 11:16:02.993579 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.993554    2552 scope.go:117] "RemoveContainer" containerID="650e3b7caf5aa12d6c4c94e9fca21ffb148d0588050ec4278d2edcf2297675be"
May 28 11:16:02.993741 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:16:02.993723    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650e3b7caf5aa12d6c4c94e9fca21ffb148d0588050ec4278d2edcf2297675be\": container with ID starting with 650e3b7caf5aa12d6c4c94e9fca21ffb148d0588050ec4278d2edcf2297675be not found: ID does not exist" containerID="650e3b7caf5aa12d6c4c94e9fca21ffb148d0588050ec4278d2edcf2297675be"
May 28 11:16:02.993786 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.993747    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650e3b7caf5aa12d6c4c94e9fca21ffb148d0588050ec4278d2edcf2297675be"} err="failed to get container status \"650e3b7caf5aa12d6c4c94e9fca21ffb148d0588050ec4278d2edcf2297675be\": rpc error: code = NotFound desc = could not find container \"650e3b7caf5aa12d6c4c94e9fca21ffb148d0588050ec4278d2edcf2297675be\": container with ID starting with 650e3b7caf5aa12d6c4c94e9fca21ffb148d0588050ec4278d2edcf2297675be not found: ID does not exist"
May 28 11:16:02.993786 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.993762    2552 scope.go:117] "RemoveContainer" containerID="4ba3f44852b6594782927b656110929953523ed2edcd667192e76740dc622790"
May 28 11:16:02.993940 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:16:02.993926    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba3f44852b6594782927b656110929953523ed2edcd667192e76740dc622790\": container with ID starting with 4ba3f44852b6594782927b656110929953523ed2edcd667192e76740dc622790 not found: ID does not exist" containerID="4ba3f44852b6594782927b656110929953523ed2edcd667192e76740dc622790"
May 28 11:16:02.993992 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.993942    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba3f44852b6594782927b656110929953523ed2edcd667192e76740dc622790"} err="failed to get container status \"4ba3f44852b6594782927b656110929953523ed2edcd667192e76740dc622790\": rpc error: code = NotFound desc = could not find container \"4ba3f44852b6594782927b656110929953523ed2edcd667192e76740dc622790\": container with ID starting with 4ba3f44852b6594782927b656110929953523ed2edcd667192e76740dc622790 not found: ID does not exist"
May 28 11:16:02.993992 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.993955    2552 scope.go:117] "RemoveContainer" containerID="acc7934fec72eba092e62a43a4fcfb24ae9109422e2b17e23cb1b2388c3e891f"
May 28 11:16:02.994092 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.993986    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m9h2\" (UniqueName: \"kubernetes.io/projected/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-kube-api-access-8m9h2\") pod \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") "
May 28 11:16:02.994092 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.994036    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-feast-data\") pod \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") "
May 28 11:16:02.994092 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.994077    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-registry-tls\") pod \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") "
May 28 11:16:02.994275 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.994169    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-online-tls\") pod \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") "
May 28 11:16:02.994275 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.994205    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-offline-tls\") pod \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") "
May 28 11:16:02.994275 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:16:02.994219    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc7934fec72eba092e62a43a4fcfb24ae9109422e2b17e23cb1b2388c3e891f\": container with ID starting with acc7934fec72eba092e62a43a4fcfb24ae9109422e2b17e23cb1b2388c3e891f not found: ID does not exist" containerID="acc7934fec72eba092e62a43a4fcfb24ae9109422e2b17e23cb1b2388c3e891f"
May 28 11:16:02.994275 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.994252    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-ui-tls\") pod \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\" (UID: \"bc7dd935-673c-4e4a-ab3a-bfcd427170ee\") "
May 28 11:16:02.994500 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.994485    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2tjm\" (UniqueName: \"kubernetes.io/projected/0b79b6bb-bd7e-443d-ad48-35d112237c97-kube-api-access-g2tjm\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:16:02.994555 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.994507    2552 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-offline-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:16:02.994555 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.994521    2552 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-ui-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:16:02.994555 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.994534    2552 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0b79b6bb-bd7e-443d-ad48-35d112237c97-online-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:16:02.994682 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.994561    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-feast-data" (OuterVolumeSpecName: "feast-data") pod "bc7dd935-673c-4e4a-ab3a-bfcd427170ee" (UID: "bc7dd935-673c-4e4a-ab3a-bfcd427170ee"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 11:16:02.994682 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.994248    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc7934fec72eba092e62a43a4fcfb24ae9109422e2b17e23cb1b2388c3e891f"} err="failed to get container status \"acc7934fec72eba092e62a43a4fcfb24ae9109422e2b17e23cb1b2388c3e891f\": rpc error: code = NotFound desc = could not find container \"acc7934fec72eba092e62a43a4fcfb24ae9109422e2b17e23cb1b2388c3e891f\": container with ID starting with acc7934fec72eba092e62a43a4fcfb24ae9109422e2b17e23cb1b2388c3e891f not found: ID does not exist"
May 28 11:16:02.994682 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.994590    2552 scope.go:117] "RemoveContainer" containerID="6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8"
May 28 11:16:02.996388 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.996364    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-online-tls" (OuterVolumeSpecName: "online-tls") pod "bc7dd935-673c-4e4a-ab3a-bfcd427170ee" (UID: "bc7dd935-673c-4e4a-ab3a-bfcd427170ee"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:16:02.996515 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.996494    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bc7dd935-673c-4e4a-ab3a-bfcd427170ee" (UID: "bc7dd935-673c-4e4a-ab3a-bfcd427170ee"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:16:02.996624 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.996605    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "bc7dd935-673c-4e4a-ab3a-bfcd427170ee" (UID: "bc7dd935-673c-4e4a-ab3a-bfcd427170ee"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:16:02.996719 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.996704    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "bc7dd935-673c-4e4a-ab3a-bfcd427170ee" (UID: "bc7dd935-673c-4e4a-ab3a-bfcd427170ee"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:16:02.996805 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:02.996786    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-kube-api-access-8m9h2" (OuterVolumeSpecName: "kube-api-access-8m9h2") pod "bc7dd935-673c-4e4a-ab3a-bfcd427170ee" (UID: "bc7dd935-673c-4e4a-ab3a-bfcd427170ee"). InnerVolumeSpecName "kube-api-access-8m9h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:16:03.002083 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.002065    2552 scope.go:117] "RemoveContainer" containerID="ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc"
May 28 11:16:03.008537 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.008521    2552 scope.go:117] "RemoveContainer" containerID="8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f"
May 28 11:16:03.014867 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.014852    2552 scope.go:117] "RemoveContainer" containerID="4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf"
May 28 11:16:03.020960 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.020940    2552 scope.go:117] "RemoveContainer" containerID="bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205"
May 28 11:16:03.027313 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.027297    2552 scope.go:117] "RemoveContainer" containerID="cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1"
May 28 11:16:03.033745 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.033731    2552 scope.go:117] "RemoveContainer" containerID="6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8"
May 28 11:16:03.033957 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:16:03.033938    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8\": container with ID starting with 6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8 not found: ID does not exist" containerID="6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8"
May 28 11:16:03.034015 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.033964    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8"} err="failed to get container status \"6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8\": rpc error: code = NotFound desc = could not find container \"6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8\": container with ID starting with 6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8 not found: ID does not exist"
May 28 11:16:03.034015 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.033981    2552 scope.go:117] "RemoveContainer" containerID="ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc"
May 28 11:16:03.034359 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:16:03.034340    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc\": container with ID starting with ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc not found: ID does not exist" containerID="ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc"
May 28 11:16:03.034406 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.034364    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc"} err="failed to get container status \"ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc\": rpc error: code = NotFound desc = could not find container \"ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc\": container with ID starting with ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc not found: ID does not exist"
May 28 11:16:03.034406 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.034378    2552 scope.go:117] "RemoveContainer" containerID="8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f"
May 28 11:16:03.034601 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:16:03.034587    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f\": container with ID starting with 8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f not found: ID does not exist" containerID="8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f"
May 28 11:16:03.034651 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.034604    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f"} err="failed to get container status \"8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f\": rpc error: code = NotFound desc = could not find container \"8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f\": container with ID starting with 8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f not found: ID does not exist"
May 28 11:16:03.034651 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.034616    2552 scope.go:117] "RemoveContainer" containerID="4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf"
May 28 11:16:03.034847 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:16:03.034833    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf\": container with ID starting with 4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf not found: ID does not exist" containerID="4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf"
May 28 11:16:03.034886 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.034849    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf"} err="failed to get container status \"4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf\": rpc error: code = NotFound desc = could not find container \"4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf\": container with ID starting with 4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf not found: ID does not exist"
May 28 11:16:03.034886 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.034861    2552 scope.go:117] "RemoveContainer" containerID="bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205"
May 28 11:16:03.035064 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:16:03.035050    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205\": container with ID starting with bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205 not found: ID does not exist" containerID="bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205"
May 28 11:16:03.035105 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.035067    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205"} err="failed to get container status \"bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205\": rpc error: code = NotFound desc = could not find container \"bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205\": container with ID starting with bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205 not found: ID does not exist"
May 28 11:16:03.035105 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.035080    2552 scope.go:117] "RemoveContainer" containerID="cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1"
May 28 11:16:03.035328 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:16:03.035310    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1\": container with ID starting with cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1 not found: ID does not exist" containerID="cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1"
May 28 11:16:03.035381 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.035334    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1"} err="failed to get container status \"cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1\": rpc error: code = NotFound desc = could not find container \"cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1\": container with ID starting with cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1 not found: ID does not exist"
May 28 11:16:03.035381 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.035349    2552 scope.go:117] "RemoveContainer" containerID="6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8"
May 28 11:16:03.035556 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.035540    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8"} err="failed to get container status \"6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8\": rpc error: code = NotFound desc = could not find container \"6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8\": container with ID starting with 6532d2722c8b8848b49c643d21299e0636abc4e6e4273956c33b906f7303dea8 not found: ID does not exist"
May 28 11:16:03.035597 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.035556    2552 scope.go:117] "RemoveContainer" containerID="ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc"
May 28 11:16:03.035767 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.035750    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc"} err="failed to get container status \"ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc\": rpc error: code = NotFound desc = could not find container \"ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc\": container with ID starting with ae1c559482285cfe70fc811692d817beba750409695dde4f1d902f079c3f4edc not found: ID does not exist"
May 28 11:16:03.035767 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.035766    2552 scope.go:117] "RemoveContainer" containerID="8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f"
May 28 11:16:03.035950 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.035927    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f"} err="failed to get container status \"8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f\": rpc error: code = NotFound desc = could not find container \"8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f\": container with ID starting with 8331d3b91c1242bb66a7c7e0e1a8f7aaf5c8527e44258a8627b605a3edc7bd2f not found: ID does not exist"
May 28 11:16:03.035950 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.035949    2552 scope.go:117] "RemoveContainer" containerID="4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf"
May 28 11:16:03.036169 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.036149    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf"} err="failed to get container status \"4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf\": rpc error: code = NotFound desc = could not find container \"4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf\": container with ID starting with 4f2474b158901317cfabdb8940d1994a877731b0ef868e7637c5512ef170a0cf not found: ID does not exist"
May 28 11:16:03.036224 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.036170    2552 scope.go:117] "RemoveContainer" containerID="bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205"
May 28 11:16:03.036359 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.036344    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205"} err="failed to get container status \"bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205\": rpc error: code = NotFound desc = could not find container \"bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205\": container with ID starting with bd14f5d5ec11c80a36ad55b273456b742a64e8a6c083ce7ca3c719b0cf4db205 not found: ID does not exist"
May 28 11:16:03.036359 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.036359    2552 scope.go:117] "RemoveContainer" containerID="cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1"
May 28 11:16:03.036556 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.036536    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1"} err="failed to get container status \"cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1\": rpc error: code = NotFound desc = could not find container \"cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1\": container with ID starting with cd913db3558b6650fa83aff096c5f589e3b66e0a91989ca69969e5749a3e95e1 not found: ID does not exist"
May 28 11:16:03.094911 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.094887    2552 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-online-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:16:03.094911 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.094909    2552 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-offline-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:16:03.095037 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.094918    2552 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-ui-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:16:03.095037 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.094927    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m9h2\" (UniqueName: \"kubernetes.io/projected/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-kube-api-access-8m9h2\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:16:03.095037 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.094937    2552 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-feast-data\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:16:03.095037 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.094945    2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bc7dd935-673c-4e4a-ab3a-bfcd427170ee-registry-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:16:03.280970 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.280944    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"]
May 28 11:16:03.284677 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:03.284654    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-8496444f68-6dnp9"]
May 28 11:16:04.517983 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:04.517945    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" path="/var/lib/kubelet/pods/0b79b6bb-bd7e-443d-ad48-35d112237c97/volumes"
May 28 11:16:04.518530 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:04.518514    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" path="/var/lib/kubelet/pods/bc7dd935-673c-4e4a-ab3a-bfcd427170ee/volumes"
May 28 11:16:15.287030 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.286994    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-vwfxl"]
May 28 11:16:15.287485 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287461    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="online"
May 28 11:16:15.287485 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287483    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="online"
May 28 11:16:15.287601 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287497    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="online"
May 28 11:16:15.287601 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287505    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="online"
May 28 11:16:15.287601 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287520    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="feast-init"
May 28 11:16:15.287601 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287529    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="feast-init"
May 28 11:16:15.287601 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287543    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="offline"
May 28 11:16:15.287601 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287551    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="offline"
May 28 11:16:15.287601 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287560    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="ui"
May 28 11:16:15.287601 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287567    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="ui"
May 28 11:16:15.287601 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287581    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="feast-apply"
May 28 11:16:15.287601 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287588    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="feast-apply"
May 28 11:16:15.287601 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287596    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="registry"
May 28 11:16:15.287601 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287603    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="registry"
May 28 11:16:15.288173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287615    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="feast-apply"
May 28 11:16:15.288173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287623    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="feast-apply"
May 28 11:16:15.288173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287634    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="feast-init"
May 28 11:16:15.288173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287642    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="feast-init"
May 28 11:16:15.288173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287657    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="offline"
May 28 11:16:15.288173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287664    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="offline"
May 28 11:16:15.288173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287673    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="ui"
May 28 11:16:15.288173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287681    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="ui"
May 28 11:16:15.288173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287761    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="online"
May 28 11:16:15.288173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287775    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="online"
May 28 11:16:15.288173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287787    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="ui"
May 28 11:16:15.288173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287800    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="registry"
May 28 11:16:15.288173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287809    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="offline"
May 28 11:16:15.288173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287821    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="0b79b6bb-bd7e-443d-ad48-35d112237c97" containerName="offline"
May 28 11:16:15.288173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.287832    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="bc7dd935-673c-4e4a-ab3a-bfcd427170ee" containerName="ui"
May 28 11:16:15.292570 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.292548    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-vwfxl"
May 28 11:16:15.294462 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.294437    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\""
May 28 11:16:15.294603 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.294512    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 11:16:15.295024 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.295009    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-zdnfh\""
May 28 11:16:15.295077 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.295033    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 11:16:15.298616 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.298594    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-vwfxl"]
May 28 11:16:15.377806 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.377773    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/a4328b8e-7ad1-4efc-b3e4-b032a26fe547-postgresdata\") pod \"postgres-7c7656cc6d-vwfxl\" (UID: \"a4328b8e-7ad1-4efc-b3e4-b032a26fe547\") " pod="test-ns-feast/postgres-7c7656cc6d-vwfxl"
May 28 11:16:15.377938 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.377873    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87glr\" (UniqueName: \"kubernetes.io/projected/a4328b8e-7ad1-4efc-b3e4-b032a26fe547-kube-api-access-87glr\") pod \"postgres-7c7656cc6d-vwfxl\" (UID: \"a4328b8e-7ad1-4efc-b3e4-b032a26fe547\") " pod="test-ns-feast/postgres-7c7656cc6d-vwfxl"
May 28 11:16:15.478775 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.478745    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/a4328b8e-7ad1-4efc-b3e4-b032a26fe547-postgresdata\") pod \"postgres-7c7656cc6d-vwfxl\" (UID: \"a4328b8e-7ad1-4efc-b3e4-b032a26fe547\") " pod="test-ns-feast/postgres-7c7656cc6d-vwfxl"
May 28 11:16:15.478954 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.478817    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-87glr\" (UniqueName: \"kubernetes.io/projected/a4328b8e-7ad1-4efc-b3e4-b032a26fe547-kube-api-access-87glr\") pod \"postgres-7c7656cc6d-vwfxl\" (UID: \"a4328b8e-7ad1-4efc-b3e4-b032a26fe547\") " pod="test-ns-feast/postgres-7c7656cc6d-vwfxl"
May 28 11:16:15.479100 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.479079    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/a4328b8e-7ad1-4efc-b3e4-b032a26fe547-postgresdata\") pod \"postgres-7c7656cc6d-vwfxl\" (UID: \"a4328b8e-7ad1-4efc-b3e4-b032a26fe547\") " pod="test-ns-feast/postgres-7c7656cc6d-vwfxl"
May 28 11:16:15.486460 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.486431    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87glr\" (UniqueName: \"kubernetes.io/projected/a4328b8e-7ad1-4efc-b3e4-b032a26fe547-kube-api-access-87glr\") pod \"postgres-7c7656cc6d-vwfxl\" (UID: \"a4328b8e-7ad1-4efc-b3e4-b032a26fe547\") " pod="test-ns-feast/postgres-7c7656cc6d-vwfxl"
May 28 11:16:15.605517 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.605434    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-vwfxl"
May 28 11:16:15.729904 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.729869    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-vwfxl"]
May 28 11:16:15.732900 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:16:15.732873    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4328b8e_7ad1_4efc_b3e4_b032a26fe547.slice/crio-62863c2fa4fedbfa3871a8f3ac401cb0769cc677ce4445800d207a1b219a618a WatchSource:0}: Error finding container 62863c2fa4fedbfa3871a8f3ac401cb0769cc677ce4445800d207a1b219a618a: Status 404 returned error can't find the container with id 62863c2fa4fedbfa3871a8f3ac401cb0769cc677ce4445800d207a1b219a618a
May 28 11:16:15.998566 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:15.998535    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-vwfxl" event={"ID":"a4328b8e-7ad1-4efc-b3e4-b032a26fe547","Type":"ContainerStarted","Data":"62863c2fa4fedbfa3871a8f3ac401cb0769cc677ce4445800d207a1b219a618a"}
May 28 11:16:21.012939 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:21.012909    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-vwfxl" event={"ID":"a4328b8e-7ad1-4efc-b3e4-b032a26fe547","Type":"ContainerStarted","Data":"f55b5ad93cef7516bbff8cada8ca4c05c2ce92eb518c6db6f191f96afedd26d1"}
May 28 11:16:21.028688 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:21.028643    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/postgres-7c7656cc6d-vwfxl" podStartSLOduration=0.860125444 podStartE2EDuration="6.028630851s" podCreationTimestamp="2026-05-28 11:16:15 +0000 UTC" firstStartedPulling="2026-05-28 11:16:15.734581634 +0000 UTC m=+791.773678595" lastFinishedPulling="2026-05-28 11:16:20.903087046 +0000 UTC m=+796.942184002" observedRunningTime="2026-05-28 11:16:21.02690134 +0000 UTC m=+797.065998318" watchObservedRunningTime="2026-05-28 11:16:21.028630851 +0000 UTC m=+797.067727831"
May 28 11:16:24.025185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.025151    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"]
May 28 11:16:24.031506 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.031487    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.033546 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.033510    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\""
May 28 11:16:24.033657 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.033616    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\""
May 28 11:16:24.033657 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.033625    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 28 11:16:24.033657 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.033616    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\""
May 28 11:16:24.033657 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.033617    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-l78n9\""
May 28 11:16:24.039502 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.039476    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"]
May 28 11:16:24.143878 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.143851    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b44b6ac7-de71-4163-9e21-050bfdb3011f-feast-data\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.144031 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.143888    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc9mx\" (UniqueName: \"kubernetes.io/projected/b44b6ac7-de71-4163-9e21-050bfdb3011f-kube-api-access-wc9mx\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.144031 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.143918    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-registry-tls\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.144031 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.144008    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-offline-tls\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.144167 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.144058    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-online-tls\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.244919 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.244891    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b44b6ac7-de71-4163-9e21-050bfdb3011f-feast-data\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.245058 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.244936    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wc9mx\" (UniqueName: \"kubernetes.io/projected/b44b6ac7-de71-4163-9e21-050bfdb3011f-kube-api-access-wc9mx\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.245058 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.244982    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-registry-tls\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.245058 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.245021    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-offline-tls\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.245248 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.245058    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-online-tls\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.245248 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:16:24.245184    2552 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found
May 28 11:16:24.245345 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:16:24.245254    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-online-tls podName:b44b6ac7-de71-4163-9e21-050bfdb3011f nodeName:}" failed. No retries permitted until 2026-05-28 11:16:24.745232557 +0000 UTC m=+800.784329527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-online-tls") pod "feast-credit-scoring-6b8dcd9c5f-48p7m" (UID: "b44b6ac7-de71-4163-9e21-050bfdb3011f") : secret "feast-credit-scoring-online-tls" not found
May 28 11:16:24.245345 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.245312    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b44b6ac7-de71-4163-9e21-050bfdb3011f-feast-data\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.247400 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.247383    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-registry-tls\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.247474 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.247406    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-offline-tls\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.252667 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.252644    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc9mx\" (UniqueName: \"kubernetes.io/projected/b44b6ac7-de71-4163-9e21-050bfdb3011f-kube-api-access-wc9mx\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.750096 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.750064    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-online-tls\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.752564 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.752534    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-online-tls\") pod \"feast-credit-scoring-6b8dcd9c5f-48p7m\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") " pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:24.941650 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:24.941624    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:25.071718 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:25.071690    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"]
May 28 11:16:25.074250 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:16:25.074221    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44b6ac7_de71_4163_9e21_050bfdb3011f.slice/crio-2c188aba85e7762aa4aba588d08d3d46e8548550c0d7ae3e16fe5039a134d43d WatchSource:0}: Error finding container 2c188aba85e7762aa4aba588d08d3d46e8548550c0d7ae3e16fe5039a134d43d: Status 404 returned error can't find the container with id 2c188aba85e7762aa4aba588d08d3d46e8548550c0d7ae3e16fe5039a134d43d
May 28 11:16:26.033842 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:26.033803    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" event={"ID":"b44b6ac7-de71-4163-9e21-050bfdb3011f","Type":"ContainerStarted","Data":"898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631"}
May 28 11:16:26.033842 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:26.033844    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" event={"ID":"b44b6ac7-de71-4163-9e21-050bfdb3011f","Type":"ContainerStarted","Data":"2c188aba85e7762aa4aba588d08d3d46e8548550c0d7ae3e16fe5039a134d43d"}
May 28 11:16:27.037786 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:27.037745    2552 generic.go:358] "Generic (PLEG): container finished" podID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerID="898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631" exitCode=0
May 28 11:16:27.038187 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:27.037797    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" event={"ID":"b44b6ac7-de71-4163-9e21-050bfdb3011f","Type":"ContainerDied","Data":"898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631"}
May 28 11:16:28.042806 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:28.042775    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" event={"ID":"b44b6ac7-de71-4163-9e21-050bfdb3011f","Type":"ContainerStarted","Data":"49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4"}
May 28 11:16:34.061602 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:34.061565    2552 generic.go:358] "Generic (PLEG): container finished" podID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerID="49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4" exitCode=0
May 28 11:16:34.062031 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:34.061608    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" event={"ID":"b44b6ac7-de71-4163-9e21-050bfdb3011f","Type":"ContainerDied","Data":"49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4"}
May 28 11:16:35.071208 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:35.071167    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" event={"ID":"b44b6ac7-de71-4163-9e21-050bfdb3011f","Type":"ContainerStarted","Data":"7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941"}
May 28 11:16:35.071208 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:35.071215    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" event={"ID":"b44b6ac7-de71-4163-9e21-050bfdb3011f","Type":"ContainerStarted","Data":"18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3"}
May 28 11:16:35.071739 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:35.071228    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" event={"ID":"b44b6ac7-de71-4163-9e21-050bfdb3011f","Type":"ContainerStarted","Data":"0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e"}
May 28 11:16:35.093662 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:35.093611    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" podStartSLOduration=11.093597984 podStartE2EDuration="11.093597984s" podCreationTimestamp="2026-05-28 11:16:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 11:16:35.092201159 +0000 UTC m=+811.131298138" watchObservedRunningTime="2026-05-28 11:16:35.093597984 +0000 UTC m=+811.132694962"
May 28 11:16:36.942318 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:36.942252    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:36.942318 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:36.942330    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:36.942858 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:36.942341    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:36.944292 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:36.944259    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.29:8816: connect: connection refused"
May 28 11:16:36.944536 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:36.944270    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused"
May 28 11:16:36.944635 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:36.944287    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 28 11:16:39.942612 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:39.942561    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused"
May 28 11:16:39.943431 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:39.943406    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:39.944059 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:39.944039    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:39.944472 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:39.944451    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:40.051202 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:40.051168    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:40.088263 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:40.088233    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:40.091314 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:40.091293    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:42.942113 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:42.942075    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused"
May 28 11:16:44.942594 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:44.942565    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:45.942606 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:45.942571    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:16:45.943426 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:16:45.943406    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:17:47.359051 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:47.359008    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"]
May 28 11:17:47.359693 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:47.359361    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="registry" containerID="cri-o://0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e" gracePeriod=30
May 28 11:17:47.359693 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:47.359398    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="offline" containerID="cri-o://7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941" gracePeriod=30
May 28 11:17:47.359693 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:47.359431    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="online" containerID="cri-o://18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3" gracePeriod=30
May 28 11:17:47.376060 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:47.376032    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-vwfxl"]
May 28 11:17:47.376258 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:47.376225    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/postgres-7c7656cc6d-vwfxl" podUID="a4328b8e-7ad1-4efc-b3e4-b032a26fe547" containerName="postgres" containerID="cri-o://f55b5ad93cef7516bbff8cada8ca4c05c2ce92eb518c6db6f191f96afedd26d1" gracePeriod=30
May 28 11:17:47.616733 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:47.616668    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-vwfxl"
May 28 11:17:47.721414 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:47.721358    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/a4328b8e-7ad1-4efc-b3e4-b032a26fe547-postgresdata\") pod \"a4328b8e-7ad1-4efc-b3e4-b032a26fe547\" (UID: \"a4328b8e-7ad1-4efc-b3e4-b032a26fe547\") "
May 28 11:17:47.721610 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:47.721422    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87glr\" (UniqueName: \"kubernetes.io/projected/a4328b8e-7ad1-4efc-b3e4-b032a26fe547-kube-api-access-87glr\") pod \"a4328b8e-7ad1-4efc-b3e4-b032a26fe547\" (UID: \"a4328b8e-7ad1-4efc-b3e4-b032a26fe547\") "
May 28 11:17:47.723469 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:47.723438    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4328b8e-7ad1-4efc-b3e4-b032a26fe547-kube-api-access-87glr" (OuterVolumeSpecName: "kube-api-access-87glr") pod "a4328b8e-7ad1-4efc-b3e4-b032a26fe547" (UID: "a4328b8e-7ad1-4efc-b3e4-b032a26fe547"). InnerVolumeSpecName "kube-api-access-87glr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:17:47.752060 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:47.752028    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4328b8e-7ad1-4efc-b3e4-b032a26fe547-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "a4328b8e-7ad1-4efc-b3e4-b032a26fe547" (UID: "a4328b8e-7ad1-4efc-b3e4-b032a26fe547"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 11:17:47.822911 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:47.822875    2552 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/a4328b8e-7ad1-4efc-b3e4-b032a26fe547-postgresdata\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:17:47.822911 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:47.822905    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87glr\" (UniqueName: \"kubernetes.io/projected/a4328b8e-7ad1-4efc-b3e4-b032a26fe547-kube-api-access-87glr\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:17:48.285552 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:48.285464    2552 generic.go:358] "Generic (PLEG): container finished" podID="a4328b8e-7ad1-4efc-b3e4-b032a26fe547" containerID="f55b5ad93cef7516bbff8cada8ca4c05c2ce92eb518c6db6f191f96afedd26d1" exitCode=0
May 28 11:17:48.285552 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:48.285513    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-vwfxl"
May 28 11:17:48.285759 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:48.285547    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-vwfxl" event={"ID":"a4328b8e-7ad1-4efc-b3e4-b032a26fe547","Type":"ContainerDied","Data":"f55b5ad93cef7516bbff8cada8ca4c05c2ce92eb518c6db6f191f96afedd26d1"}
May 28 11:17:48.285759 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:48.285586    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-vwfxl" event={"ID":"a4328b8e-7ad1-4efc-b3e4-b032a26fe547","Type":"ContainerDied","Data":"62863c2fa4fedbfa3871a8f3ac401cb0769cc677ce4445800d207a1b219a618a"}
May 28 11:17:48.285759 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:48.285600    2552 scope.go:117] "RemoveContainer" containerID="f55b5ad93cef7516bbff8cada8ca4c05c2ce92eb518c6db6f191f96afedd26d1"
May 28 11:17:48.287979 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:48.287948    2552 generic.go:358] "Generic (PLEG): container finished" podID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerID="18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3" exitCode=0
May 28 11:17:48.288095 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:48.287992    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" event={"ID":"b44b6ac7-de71-4163-9e21-050bfdb3011f","Type":"ContainerDied","Data":"18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3"}
May 28 11:17:48.293690 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:48.293672    2552 scope.go:117] "RemoveContainer" containerID="f55b5ad93cef7516bbff8cada8ca4c05c2ce92eb518c6db6f191f96afedd26d1"
May 28 11:17:48.293991 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:17:48.293972    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f55b5ad93cef7516bbff8cada8ca4c05c2ce92eb518c6db6f191f96afedd26d1\": container with ID starting with f55b5ad93cef7516bbff8cada8ca4c05c2ce92eb518c6db6f191f96afedd26d1 not found: ID does not exist" containerID="f55b5ad93cef7516bbff8cada8ca4c05c2ce92eb518c6db6f191f96afedd26d1"
May 28 11:17:48.294079 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:48.293999    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f55b5ad93cef7516bbff8cada8ca4c05c2ce92eb518c6db6f191f96afedd26d1"} err="failed to get container status \"f55b5ad93cef7516bbff8cada8ca4c05c2ce92eb518c6db6f191f96afedd26d1\": rpc error: code = NotFound desc = could not find container \"f55b5ad93cef7516bbff8cada8ca4c05c2ce92eb518c6db6f191f96afedd26d1\": container with ID starting with f55b5ad93cef7516bbff8cada8ca4c05c2ce92eb518c6db6f191f96afedd26d1 not found: ID does not exist"
May 28 11:17:48.307636 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:48.307614    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-vwfxl"]
May 28 11:17:48.314016 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:48.313997    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-vwfxl"]
May 28 11:17:48.518147 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:48.518096    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4328b8e-7ad1-4efc-b3e4-b032a26fe547" path="/var/lib/kubelet/pods/a4328b8e-7ad1-4efc-b3e4-b032a26fe547/volumes"
May 28 11:17:50.088474 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:17:50.088422    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 28 11:18:00.088675 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:00.088629    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 28 11:18:04.462184 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:04.462111    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:18:04.464387 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:04.464364    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:18:10.093628 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:10.093585    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused"
May 28 11:18:10.095148 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:10.093900    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:18:18.001946 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.001920    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:18:18.173330 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.173260    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b44b6ac7-de71-4163-9e21-050bfdb3011f-feast-data\") pod \"b44b6ac7-de71-4163-9e21-050bfdb3011f\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") "
May 28 11:18:18.173330 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.173295    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-offline-tls\") pod \"b44b6ac7-de71-4163-9e21-050bfdb3011f\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") "
May 28 11:18:18.173526 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.173350    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-online-tls\") pod \"b44b6ac7-de71-4163-9e21-050bfdb3011f\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") "
May 28 11:18:18.173526 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.173377    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-registry-tls\") pod \"b44b6ac7-de71-4163-9e21-050bfdb3011f\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") "
May 28 11:18:18.173526 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.173404    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc9mx\" (UniqueName: \"kubernetes.io/projected/b44b6ac7-de71-4163-9e21-050bfdb3011f-kube-api-access-wc9mx\") pod \"b44b6ac7-de71-4163-9e21-050bfdb3011f\" (UID: \"b44b6ac7-de71-4163-9e21-050bfdb3011f\") "
May 28 11:18:18.175491 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.175464    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b44b6ac7-de71-4163-9e21-050bfdb3011f" (UID: "b44b6ac7-de71-4163-9e21-050bfdb3011f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:18:18.175491 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.175487    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "b44b6ac7-de71-4163-9e21-050bfdb3011f" (UID: "b44b6ac7-de71-4163-9e21-050bfdb3011f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:18:18.175640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.175496    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44b6ac7-de71-4163-9e21-050bfdb3011f-kube-api-access-wc9mx" (OuterVolumeSpecName: "kube-api-access-wc9mx") pod "b44b6ac7-de71-4163-9e21-050bfdb3011f" (UID: "b44b6ac7-de71-4163-9e21-050bfdb3011f"). InnerVolumeSpecName "kube-api-access-wc9mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:18:18.175851 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.175829    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-online-tls" (OuterVolumeSpecName: "online-tls") pod "b44b6ac7-de71-4163-9e21-050bfdb3011f" (UID: "b44b6ac7-de71-4163-9e21-050bfdb3011f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:18:18.181353 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.181326    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44b6ac7-de71-4163-9e21-050bfdb3011f-feast-data" (OuterVolumeSpecName: "feast-data") pod "b44b6ac7-de71-4163-9e21-050bfdb3011f" (UID: "b44b6ac7-de71-4163-9e21-050bfdb3011f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 11:18:18.274684 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.274662    2552 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-offline-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:18:18.274684 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.274684    2552 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-online-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:18:18.274834 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.274693    2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b44b6ac7-de71-4163-9e21-050bfdb3011f-registry-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:18:18.274834 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.274701    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc9mx\" (UniqueName: \"kubernetes.io/projected/b44b6ac7-de71-4163-9e21-050bfdb3011f-kube-api-access-wc9mx\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:18:18.274834 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.274711    2552 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b44b6ac7-de71-4163-9e21-050bfdb3011f-feast-data\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:18:18.381045 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.381014    2552 generic.go:358] "Generic (PLEG): container finished" podID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerID="7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941" exitCode=137
May 28 11:18:18.381045 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.381037    2552 generic.go:358] "Generic (PLEG): container finished" podID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerID="0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e" exitCode=137
May 28 11:18:18.381255 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.381050    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" event={"ID":"b44b6ac7-de71-4163-9e21-050bfdb3011f","Type":"ContainerDied","Data":"7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941"}
May 28 11:18:18.381255 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.381092    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" event={"ID":"b44b6ac7-de71-4163-9e21-050bfdb3011f","Type":"ContainerDied","Data":"0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e"}
May 28 11:18:18.381255 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.381104    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m" event={"ID":"b44b6ac7-de71-4163-9e21-050bfdb3011f","Type":"ContainerDied","Data":"2c188aba85e7762aa4aba588d08d3d46e8548550c0d7ae3e16fe5039a134d43d"}
May 28 11:18:18.381255 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.381112    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"
May 28 11:18:18.381255 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.381117    2552 scope.go:117] "RemoveContainer" containerID="7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941"
May 28 11:18:18.389182 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.389164    2552 scope.go:117] "RemoveContainer" containerID="18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3"
May 28 11:18:18.396243 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.396227    2552 scope.go:117] "RemoveContainer" containerID="0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e"
May 28 11:18:18.403292 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.403273    2552 scope.go:117] "RemoveContainer" containerID="49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4"
May 28 11:18:18.404761 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.404741    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"]
May 28 11:18:18.410617 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.410560    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-6b8dcd9c5f-48p7m"]
May 28 11:18:18.410670 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.410646    2552 scope.go:117] "RemoveContainer" containerID="898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631"
May 28 11:18:18.417281 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.417267    2552 scope.go:117] "RemoveContainer" containerID="7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941"
May 28 11:18:18.417492 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:18:18.417474    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941\": container with ID starting with 7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941 not found: ID does not exist" containerID="7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941"
May 28 11:18:18.417536 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.417500    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941"} err="failed to get container status \"7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941\": rpc error: code = NotFound desc = could not find container \"7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941\": container with ID starting with 7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941 not found: ID does not exist"
May 28 11:18:18.417536 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.417519    2552 scope.go:117] "RemoveContainer" containerID="18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3"
May 28 11:18:18.417730 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:18:18.417715    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3\": container with ID starting with 18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3 not found: ID does not exist" containerID="18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3"
May 28 11:18:18.417776 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.417733    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3"} err="failed to get container status \"18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3\": rpc error: code = NotFound desc = could not find container \"18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3\": container with ID starting with 18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3 not found: ID does not exist"
May 28 11:18:18.417776 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.417748    2552 scope.go:117] "RemoveContainer" containerID="0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e"
May 28 11:18:18.418016 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:18:18.417997    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e\": container with ID starting with 0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e not found: ID does not exist" containerID="0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e"
May 28 11:18:18.418092 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.418025    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e"} err="failed to get container status \"0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e\": rpc error: code = NotFound desc = could not find container \"0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e\": container with ID starting with 0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e not found: ID does not exist"
May 28 11:18:18.418092 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.418048    2552 scope.go:117] "RemoveContainer" containerID="49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4"
May 28 11:18:18.418299 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:18:18.418283    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4\": container with ID starting with 49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4 not found: ID does not exist" containerID="49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4"
May 28 11:18:18.418352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.418305    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4"} err="failed to get container status \"49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4\": rpc error: code = NotFound desc = could not find container \"49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4\": container with ID starting with 49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4 not found: ID does not exist"
May 28 11:18:18.418352 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.418321    2552 scope.go:117] "RemoveContainer" containerID="898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631"
May 28 11:18:18.418516 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:18:18.418500    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631\": container with ID starting with 898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631 not found: ID does not exist" containerID="898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631"
May 28 11:18:18.418559 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.418519    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631"} err="failed to get container status \"898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631\": rpc error: code = NotFound desc = could not find container \"898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631\": container with ID starting with 898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631 not found: ID does not exist"
May 28 11:18:18.418559 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.418539    2552 scope.go:117] "RemoveContainer" containerID="7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941"
May 28 11:18:18.418729 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.418710    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941"} err="failed to get container status \"7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941\": rpc error: code = NotFound desc = could not find container \"7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941\": container with ID starting with 7e8953f48aa39e1118c9322651e3c43d5f4f7fe577d12b95b5aef6d0b7f91941 not found: ID does not exist"
May 28 11:18:18.418798 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.418731    2552 scope.go:117] "RemoveContainer" containerID="18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3"
May 28 11:18:18.418932 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.418912    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3"} err="failed to get container status \"18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3\": rpc error: code = NotFound desc = could not find container \"18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3\": container with ID starting with 18efed90c2cd70ea04c85abc78abd3dfc3d38dba879614b556921150feb414b3 not found: ID does not exist"
May 28 11:18:18.418978 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.418935    2552 scope.go:117] "RemoveContainer" containerID="0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e"
May 28 11:18:18.419236 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.419118    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e"} err="failed to get container status \"0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e\": rpc error: code = NotFound desc = could not find container \"0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e\": container with ID starting with 0c4aafd3ed8c304cc8b0558c16b195dc42194982e65896cdc7b9f18df5063f8e not found: ID does not exist"
May 28 11:18:18.419289 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.419241    2552 scope.go:117] "RemoveContainer" containerID="49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4"
May 28 11:18:18.419464 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.419445    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4"} err="failed to get container status \"49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4\": rpc error: code = NotFound desc = could not find container \"49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4\": container with ID starting with 49464236d4a8756adb965c68cfd5317796edc916e5c454cbe0079ed04a971dd4 not found: ID does not exist"
May 28 11:18:18.419536 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.419466    2552 scope.go:117] "RemoveContainer" containerID="898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631"
May 28 11:18:18.419698 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.419679    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631"} err="failed to get container status \"898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631\": rpc error: code = NotFound desc = could not find container \"898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631\": container with ID starting with 898b1389f4630ae396464708667a3a98129f5b149831da6533a28492a5c9b631 not found: ID does not exist"
May 28 11:18:18.517113 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:18:18.517089    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" path="/var/lib/kubelet/pods/b44b6ac7-de71-4163-9e21-050bfdb3011f/volumes"
May 28 11:20:13.869611 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.869578    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"]
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.869881    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="registry"
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.869891    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="registry"
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.869903    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="online"
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.869909    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="online"
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.869926    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="feast-apply"
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.869931    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="feast-apply"
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.869940    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4328b8e-7ad1-4efc-b3e4-b032a26fe547" containerName="postgres"
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.869945    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4328b8e-7ad1-4efc-b3e4-b032a26fe547" containerName="postgres"
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.869951    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="feast-init"
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.869956    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="feast-init"
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.869965    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="offline"
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.869970    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="offline"
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.870011    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="registry"
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.870018    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="online"
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.870027    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="a4328b8e-7ad1-4efc-b3e4-b032a26fe547" containerName="postgres"
May 28 11:20:13.870021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.870035    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="b44b6ac7-de71-4163-9e21-050bfdb3011f" containerName="offline"
May 28 11:20:13.871914 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.871898    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"
May 28 11:20:13.873957 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.873934    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\""
May 28 11:20:13.874234 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.874216    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\""
May 28 11:20:13.874607 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.874583    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\""
May 28 11:20:13.874717 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.874592    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-56r8m\""
May 28 11:20:13.878284 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.878262    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"]
May 28 11:20:13.913473 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.913449    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b875cbdc-d394-437a-85bd-a4ee017d217a-postgresdata\") pod \"postgres-7c7656cc6d-8gg9z\" (UID: \"b875cbdc-d394-437a-85bd-a4ee017d217a\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"
May 28 11:20:13.913576 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:13.913477    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7n88\" (UniqueName: \"kubernetes.io/projected/b875cbdc-d394-437a-85bd-a4ee017d217a-kube-api-access-v7n88\") pod \"postgres-7c7656cc6d-8gg9z\" (UID: \"b875cbdc-d394-437a-85bd-a4ee017d217a\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"
May 28 11:20:14.014619 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:14.014593    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b875cbdc-d394-437a-85bd-a4ee017d217a-postgresdata\") pod \"postgres-7c7656cc6d-8gg9z\" (UID: \"b875cbdc-d394-437a-85bd-a4ee017d217a\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"
May 28 11:20:14.014779 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:14.014630    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v7n88\" (UniqueName: \"kubernetes.io/projected/b875cbdc-d394-437a-85bd-a4ee017d217a-kube-api-access-v7n88\") pod \"postgres-7c7656cc6d-8gg9z\" (UID: \"b875cbdc-d394-437a-85bd-a4ee017d217a\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"
May 28 11:20:14.014957 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:14.014937    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b875cbdc-d394-437a-85bd-a4ee017d217a-postgresdata\") pod \"postgres-7c7656cc6d-8gg9z\" (UID: \"b875cbdc-d394-437a-85bd-a4ee017d217a\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"
May 28 11:20:14.022641 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:14.022620    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7n88\" (UniqueName: \"kubernetes.io/projected/b875cbdc-d394-437a-85bd-a4ee017d217a-kube-api-access-v7n88\") pod \"postgres-7c7656cc6d-8gg9z\" (UID: \"b875cbdc-d394-437a-85bd-a4ee017d217a\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"
May 28 11:20:14.182659 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:14.182586    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"
May 28 11:20:14.303046 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:14.303013    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"]
May 28 11:20:14.305881 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:20:14.305859    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb875cbdc_d394_437a_85bd_a4ee017d217a.slice/crio-24fb15fbe01243d57efe78d24808576cce591afa563d1fbdfcf71dc0680d135e WatchSource:0}: Error finding container 24fb15fbe01243d57efe78d24808576cce591afa563d1fbdfcf71dc0680d135e: Status 404 returned error can't find the container with id 24fb15fbe01243d57efe78d24808576cce591afa563d1fbdfcf71dc0680d135e
May 28 11:20:14.307639 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:14.307622    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 11:20:14.695843 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:14.695804    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z" event={"ID":"b875cbdc-d394-437a-85bd-a4ee017d217a","Type":"ContainerStarted","Data":"ae56564b8fe9e0050c0100b9797fd18a4888e4d1641c33fdbb6046b73aecf96b"}
May 28 11:20:14.695843 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:14.695843    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z" event={"ID":"b875cbdc-d394-437a-85bd-a4ee017d217a","Type":"ContainerStarted","Data":"24fb15fbe01243d57efe78d24808576cce591afa563d1fbdfcf71dc0680d135e"}
May 28 11:20:14.710174 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:14.710112    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z" podStartSLOduration=1.71009916 podStartE2EDuration="1.71009916s" podCreationTimestamp="2026-05-28 11:20:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 11:20:14.708726684 +0000 UTC m=+1030.747823664" watchObservedRunningTime="2026-05-28 11:20:14.71009916 +0000 UTC m=+1030.749196138"
May 28 11:20:19.521310 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.521271    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"]
May 28 11:20:19.523754 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.523738    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:19.526001 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.525971    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\""
May 28 11:20:19.526155 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.525973    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\""
May 28 11:20:19.526155 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.525974    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-qcz2n\""
May 28 11:20:19.526155 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.525977    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\""
May 28 11:20:19.533080 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.533057    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"]
May 28 11:20:19.554217 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.554191    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-feast-data\") pod \"feast-credit-scoring-844579f4c-kz5lw\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") " pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:19.554311 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.554218    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pm5m\" (UniqueName: \"kubernetes.io/projected/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-kube-api-access-5pm5m\") pod \"feast-credit-scoring-844579f4c-kz5lw\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") " pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:19.554311 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.554245    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-registry-tls\") pod \"feast-credit-scoring-844579f4c-kz5lw\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") " pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:19.554380 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.554320    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-online-tls\") pod \"feast-credit-scoring-844579f4c-kz5lw\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") " pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:19.655019 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.654991    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-online-tls\") pod \"feast-credit-scoring-844579f4c-kz5lw\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") " pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:19.655152 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.655043    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-feast-data\") pod \"feast-credit-scoring-844579f4c-kz5lw\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") " pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:19.655152 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.655063    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5pm5m\" (UniqueName: \"kubernetes.io/projected/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-kube-api-access-5pm5m\") pod \"feast-credit-scoring-844579f4c-kz5lw\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") " pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:19.655152 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.655100    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-registry-tls\") pod \"feast-credit-scoring-844579f4c-kz5lw\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") " pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:19.655484 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.655465    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-feast-data\") pod \"feast-credit-scoring-844579f4c-kz5lw\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") " pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:19.657483 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.657462    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-online-tls\") pod \"feast-credit-scoring-844579f4c-kz5lw\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") " pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:19.657559 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.657545    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-registry-tls\") pod \"feast-credit-scoring-844579f4c-kz5lw\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") " pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:19.661987 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.661966    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pm5m\" (UniqueName: \"kubernetes.io/projected/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-kube-api-access-5pm5m\") pod \"feast-credit-scoring-844579f4c-kz5lw\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") " pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:19.833752 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.833684    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:19.962893 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:19.962866    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"]
May 28 11:20:19.965415 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:20:19.965382    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c4fd09_3cf0_4f96_8abd_20426b8b0f0d.slice/crio-d851ba3120dca516e0c6798de4b36e29b19f4117c742565e3ace49281af27516 WatchSource:0}: Error finding container d851ba3120dca516e0c6798de4b36e29b19f4117c742565e3ace49281af27516: Status 404 returned error can't find the container with id d851ba3120dca516e0c6798de4b36e29b19f4117c742565e3ace49281af27516
May 28 11:20:20.714912 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:20.714880    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" event={"ID":"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d","Type":"ContainerStarted","Data":"f3dd4bcd98380ff66bf2b6be7dc516e778c9bb4ef7594fca58b1ecb6e291f646"}
May 28 11:20:20.714912 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:20.714915    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" event={"ID":"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d","Type":"ContainerStarted","Data":"d851ba3120dca516e0c6798de4b36e29b19f4117c742565e3ace49281af27516"}
May 28 11:20:21.721217 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:21.721177    2552 generic.go:358] "Generic (PLEG): container finished" podID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerID="f3dd4bcd98380ff66bf2b6be7dc516e778c9bb4ef7594fca58b1ecb6e291f646" exitCode=0
May 28 11:20:21.721678 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:21.721259    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" event={"ID":"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d","Type":"ContainerDied","Data":"f3dd4bcd98380ff66bf2b6be7dc516e778c9bb4ef7594fca58b1ecb6e291f646"}
May 28 11:20:22.725009 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:22.724974    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" event={"ID":"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d","Type":"ContainerStarted","Data":"27ea1576395299c1ff4c3d964fecc4af32dfead149680fac877c1cb6014585a2"}
May 28 11:20:28.741791 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:28.741763    2552 generic.go:358] "Generic (PLEG): container finished" podID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerID="27ea1576395299c1ff4c3d964fecc4af32dfead149680fac877c1cb6014585a2" exitCode=0
May 28 11:20:28.742154 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:28.741815    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" event={"ID":"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d","Type":"ContainerDied","Data":"27ea1576395299c1ff4c3d964fecc4af32dfead149680fac877c1cb6014585a2"}
May 28 11:20:29.747187 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:29.747153    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" event={"ID":"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d","Type":"ContainerStarted","Data":"484c7238d7ba0f929ccfff12b52ab81192187bb684fbea3beee86f9001c42094"}
May 28 11:20:29.747187 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:29.747192    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" event={"ID":"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d","Type":"ContainerStarted","Data":"4297fb7119c4c428a1910d54239124a9019156f496727074ed32e42fd61f3344"}
May 28 11:20:29.770551 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:29.770497    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" podStartSLOduration=10.770478666 podStartE2EDuration="10.770478666s" podCreationTimestamp="2026-05-28 11:20:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 11:20:29.768218328 +0000 UTC m=+1045.807315322" watchObservedRunningTime="2026-05-28 11:20:29.770478666 +0000 UTC m=+1045.809575648"
May 28 11:20:29.834827 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:29.834803    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:29.834827 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:29.834834    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:31.834520 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:31.834491    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:31.834904 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:31.834530    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:31.836170 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:31.836119    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused"
May 28 11:20:31.836170 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:31.836148    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused"
May 28 11:20:34.834889 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:34.834842    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused"
May 28 11:20:34.842117 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:34.842086    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:34.846173 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:34.846154    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:37.834861 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:37.834808    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused"
May 28 11:20:40.835315 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:40.835280    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:40.836098 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:40.836078    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:20:45.356342 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.356303    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"]
May 28 11:20:45.361630 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.361612    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:20:45.363817 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.363792    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\""
May 28 11:20:45.363969 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.363948    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-bh2p5\""
May 28 11:20:45.364048 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.364032    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\""
May 28 11:20:45.364105 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.364040    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\""
May 28 11:20:45.370608 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.370581    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"]
May 28 11:20:45.461453 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.461422    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0da1cf97-4702-4271-b2ca-edb6bdad80d7-online-tls\") pod \"feast-driver-ranking-bfdb7cf9b-q86zz\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") " pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:20:45.461584 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.461508    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0da1cf97-4702-4271-b2ca-edb6bdad80d7-registry-tls\") pod \"feast-driver-ranking-bfdb7cf9b-q86zz\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") " pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:20:45.461584 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.461540    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjkpp\" (UniqueName: \"kubernetes.io/projected/0da1cf97-4702-4271-b2ca-edb6bdad80d7-kube-api-access-bjkpp\") pod \"feast-driver-ranking-bfdb7cf9b-q86zz\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") " pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:20:45.461584 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.461574    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0da1cf97-4702-4271-b2ca-edb6bdad80d7-feast-data\") pod \"feast-driver-ranking-bfdb7cf9b-q86zz\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") " pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:20:45.562652 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.562630    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0da1cf97-4702-4271-b2ca-edb6bdad80d7-registry-tls\") pod \"feast-driver-ranking-bfdb7cf9b-q86zz\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") " pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:20:45.562776 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.562662    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bjkpp\" (UniqueName: \"kubernetes.io/projected/0da1cf97-4702-4271-b2ca-edb6bdad80d7-kube-api-access-bjkpp\") pod \"feast-driver-ranking-bfdb7cf9b-q86zz\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") " pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:20:45.562776 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.562686    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0da1cf97-4702-4271-b2ca-edb6bdad80d7-feast-data\") pod \"feast-driver-ranking-bfdb7cf9b-q86zz\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") " pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:20:45.562776 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.562712    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0da1cf97-4702-4271-b2ca-edb6bdad80d7-online-tls\") pod \"feast-driver-ranking-bfdb7cf9b-q86zz\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") " pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:20:45.563061 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.563042    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0da1cf97-4702-4271-b2ca-edb6bdad80d7-feast-data\") pod \"feast-driver-ranking-bfdb7cf9b-q86zz\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") " pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:20:45.565212 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.565182    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0da1cf97-4702-4271-b2ca-edb6bdad80d7-online-tls\") pod \"feast-driver-ranking-bfdb7cf9b-q86zz\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") " pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:20:45.565315 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.565232    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0da1cf97-4702-4271-b2ca-edb6bdad80d7-registry-tls\") pod \"feast-driver-ranking-bfdb7cf9b-q86zz\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") " pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:20:45.570088 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.570072    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjkpp\" (UniqueName: \"kubernetes.io/projected/0da1cf97-4702-4271-b2ca-edb6bdad80d7-kube-api-access-bjkpp\") pod \"feast-driver-ranking-bfdb7cf9b-q86zz\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") " pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:20:45.671617 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.671556    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:20:45.796849 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:45.796810    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"]
May 28 11:20:45.800520 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:20:45.800489    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da1cf97_4702_4271_b2ca_edb6bdad80d7.slice/crio-feeca8e720a209f3fa54f774729f2db920c95c84de045458fa173e45be396237 WatchSource:0}: Error finding container feeca8e720a209f3fa54f774729f2db920c95c84de045458fa173e45be396237: Status 404 returned error can't find the container with id feeca8e720a209f3fa54f774729f2db920c95c84de045458fa173e45be396237
May 28 11:20:46.799673 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:46.799636    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" event={"ID":"0da1cf97-4702-4271-b2ca-edb6bdad80d7","Type":"ContainerStarted","Data":"bc5bdf494adac3e20c016114ac6511c4af909719f273155986c762c12b3c4675"}
May 28 11:20:46.799673 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:46.799674    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" event={"ID":"0da1cf97-4702-4271-b2ca-edb6bdad80d7","Type":"ContainerStarted","Data":"feeca8e720a209f3fa54f774729f2db920c95c84de045458fa173e45be396237"}
May 28 11:20:50.810991 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:50.810957    2552 generic.go:358] "Generic (PLEG): container finished" podID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerID="bc5bdf494adac3e20c016114ac6511c4af909719f273155986c762c12b3c4675" exitCode=0
May 28 11:20:50.811361 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:50.811029    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" event={"ID":"0da1cf97-4702-4271-b2ca-edb6bdad80d7","Type":"ContainerDied","Data":"bc5bdf494adac3e20c016114ac6511c4af909719f273155986c762c12b3c4675"}
May 28 11:20:51.816253 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:51.816218    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" event={"ID":"0da1cf97-4702-4271-b2ca-edb6bdad80d7","Type":"ContainerStarted","Data":"4fece75cc421c78cf1bf5e7ff24d293a9cd019b815d4a59a6139798b98c80955"}
May 28 11:20:57.836408 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:57.836324    2552 generic.go:358] "Generic (PLEG): container finished" podID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerID="4fece75cc421c78cf1bf5e7ff24d293a9cd019b815d4a59a6139798b98c80955" exitCode=0
May 28 11:20:57.836408 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:57.836387    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" event={"ID":"0da1cf97-4702-4271-b2ca-edb6bdad80d7","Type":"ContainerDied","Data":"4fece75cc421c78cf1bf5e7ff24d293a9cd019b815d4a59a6139798b98c80955"}
May 28 11:20:58.841794 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:58.841759    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" event={"ID":"0da1cf97-4702-4271-b2ca-edb6bdad80d7","Type":"ContainerStarted","Data":"a876019623ac647c7ea2bf99e6c211fae304bf86e70cf33830352f7c0111b333"}
May 28 11:20:58.841794 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:58.841797    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" event={"ID":"0da1cf97-4702-4271-b2ca-edb6bdad80d7","Type":"ContainerStarted","Data":"f7669d141460c23ca3ac356e05a0464ba94e1a2d1eb18e3241689cf0995729cd"}
May 28 11:20:58.862760 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:20:58.862708    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" podStartSLOduration=13.862694654 podStartE2EDuration="13.862694654s" podCreationTimestamp="2026-05-28 11:20:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 11:20:58.861454486 +0000 UTC m=+1074.900551465" watchObservedRunningTime="2026-05-28 11:20:58.862694654 +0000 UTC m=+1074.901791633"
May 28 11:21:00.671888 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:21:00.671860    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:21:00.671888 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:21:00.671895    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:21:00.673632 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:21:00.673599    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused"
May 28 11:21:00.673632 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:21:00.673617    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused"
May 28 11:21:03.672061 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:21:03.672017    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused"
May 28 11:21:03.908916 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:21:03.908880    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:21:03.909423 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:21:03.909398    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:21:03.912231 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:21:03.912208    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:21:05.672411 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:21:05.672370    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:21:06.672321 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:21:06.672285    2552 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.32:6571: connect: connection refused"
May 28 11:21:09.672717 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:21:09.672682    2552 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:21:09.673328 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:21:09.673311    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:22:24.815329 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:24.815241    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"]
May 28 11:22:24.815891 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:24.815552    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="registry" containerID="cri-o://4297fb7119c4c428a1910d54239124a9019156f496727074ed32e42fd61f3344" gracePeriod=30
May 28 11:22:24.815891 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:24.815634    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="online" containerID="cri-o://484c7238d7ba0f929ccfff12b52ab81192187bb684fbea3beee86f9001c42094" gracePeriod=30
May 28 11:22:24.819879 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:24.819851    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"]
May 28 11:22:24.820217 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:24.820192    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="registry" containerID="cri-o://f7669d141460c23ca3ac356e05a0464ba94e1a2d1eb18e3241689cf0995729cd" gracePeriod=30
May 28 11:22:24.820437 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:24.820221    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="online" containerID="cri-o://a876019623ac647c7ea2bf99e6c211fae304bf86e70cf33830352f7c0111b333" gracePeriod=30
May 28 11:22:26.140443 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:26.140410    2552 generic.go:358] "Generic (PLEG): container finished" podID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerID="484c7238d7ba0f929ccfff12b52ab81192187bb684fbea3beee86f9001c42094" exitCode=0
May 28 11:22:26.140863 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:26.140477    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" event={"ID":"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d","Type":"ContainerDied","Data":"484c7238d7ba0f929ccfff12b52ab81192187bb684fbea3beee86f9001c42094"}
May 28 11:22:26.142439 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:26.142416    2552 generic.go:358] "Generic (PLEG): container finished" podID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerID="a876019623ac647c7ea2bf99e6c211fae304bf86e70cf33830352f7c0111b333" exitCode=0
May 28 11:22:26.142564 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:26.142480    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" event={"ID":"0da1cf97-4702-4271-b2ca-edb6bdad80d7","Type":"ContainerDied","Data":"a876019623ac647c7ea2bf99e6c211fae304bf86e70cf33830352f7c0111b333"}
May 28 11:22:26.765408 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:26.765376    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"]
May 28 11:22:26.765586 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:26.765565    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z" podUID="b875cbdc-d394-437a-85bd-a4ee017d217a" containerName="postgres" containerID="cri-o://ae56564b8fe9e0050c0100b9797fd18a4888e4d1641c33fdbb6046b73aecf96b" gracePeriod=30
May 28 11:22:26.996807 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:26.996786    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"
May 28 11:22:27.082594 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:27.082509    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7n88\" (UniqueName: \"kubernetes.io/projected/b875cbdc-d394-437a-85bd-a4ee017d217a-kube-api-access-v7n88\") pod \"b875cbdc-d394-437a-85bd-a4ee017d217a\" (UID: \"b875cbdc-d394-437a-85bd-a4ee017d217a\") "
May 28 11:22:27.082594 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:27.082552    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b875cbdc-d394-437a-85bd-a4ee017d217a-postgresdata\") pod \"b875cbdc-d394-437a-85bd-a4ee017d217a\" (UID: \"b875cbdc-d394-437a-85bd-a4ee017d217a\") "
May 28 11:22:27.084747 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:27.084711    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b875cbdc-d394-437a-85bd-a4ee017d217a-kube-api-access-v7n88" (OuterVolumeSpecName: "kube-api-access-v7n88") pod "b875cbdc-d394-437a-85bd-a4ee017d217a" (UID: "b875cbdc-d394-437a-85bd-a4ee017d217a"). InnerVolumeSpecName "kube-api-access-v7n88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:22:27.113617 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:27.113590    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b875cbdc-d394-437a-85bd-a4ee017d217a-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "b875cbdc-d394-437a-85bd-a4ee017d217a" (UID: "b875cbdc-d394-437a-85bd-a4ee017d217a"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 11:22:27.146371 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:27.146345    2552 generic.go:358] "Generic (PLEG): container finished" podID="b875cbdc-d394-437a-85bd-a4ee017d217a" containerID="ae56564b8fe9e0050c0100b9797fd18a4888e4d1641c33fdbb6046b73aecf96b" exitCode=0
May 28 11:22:27.146692 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:27.146391    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"
May 28 11:22:27.146692 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:27.146398    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z" event={"ID":"b875cbdc-d394-437a-85bd-a4ee017d217a","Type":"ContainerDied","Data":"ae56564b8fe9e0050c0100b9797fd18a4888e4d1641c33fdbb6046b73aecf96b"}
May 28 11:22:27.146692 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:27.146435    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-8gg9z" event={"ID":"b875cbdc-d394-437a-85bd-a4ee017d217a","Type":"ContainerDied","Data":"24fb15fbe01243d57efe78d24808576cce591afa563d1fbdfcf71dc0680d135e"}
May 28 11:22:27.146692 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:27.146451    2552 scope.go:117] "RemoveContainer" containerID="ae56564b8fe9e0050c0100b9797fd18a4888e4d1641c33fdbb6046b73aecf96b"
May 28 11:22:27.154844 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:27.154826    2552 scope.go:117] "RemoveContainer" containerID="ae56564b8fe9e0050c0100b9797fd18a4888e4d1641c33fdbb6046b73aecf96b"
May 28 11:22:27.155026 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:22:27.155010    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae56564b8fe9e0050c0100b9797fd18a4888e4d1641c33fdbb6046b73aecf96b\": container with ID starting with ae56564b8fe9e0050c0100b9797fd18a4888e4d1641c33fdbb6046b73aecf96b not found: ID does not exist" containerID="ae56564b8fe9e0050c0100b9797fd18a4888e4d1641c33fdbb6046b73aecf96b"
May 28 11:22:27.155083 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:27.155034    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae56564b8fe9e0050c0100b9797fd18a4888e4d1641c33fdbb6046b73aecf96b"} err="failed to get container status \"ae56564b8fe9e0050c0100b9797fd18a4888e4d1641c33fdbb6046b73aecf96b\": rpc error: code = NotFound desc = could not find container \"ae56564b8fe9e0050c0100b9797fd18a4888e4d1641c33fdbb6046b73aecf96b\": container with ID starting with ae56564b8fe9e0050c0100b9797fd18a4888e4d1641c33fdbb6046b73aecf96b not found: ID does not exist"
May 28 11:22:27.165840 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:27.165816    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"]
May 28 11:22:27.169403 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:27.169383    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-8gg9z"]
May 28 11:22:27.183444 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:27.183426    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7n88\" (UniqueName: \"kubernetes.io/projected/b875cbdc-d394-437a-85bd-a4ee017d217a-kube-api-access-v7n88\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:22:27.183510 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:27.183447    2552 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b875cbdc-d394-437a-85bd-a4ee017d217a-postgresdata\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:22:28.517657 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:28.517617    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b875cbdc-d394-437a-85bd-a4ee017d217a" path="/var/lib/kubelet/pods/b875cbdc-d394-437a-85bd-a4ee017d217a/volumes"
May 28 11:22:33.910179 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:33.910118    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused"
May 28 11:22:34.843389 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:34.843350    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused"
May 28 11:22:43.910477 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:43.910436    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused"
May 28 11:22:44.843032 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:44.842989    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused"
May 28 11:22:53.909783 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:53.909733    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="online" probeResult="failure" output="Get \"https://10.134.0.32:6567/health\": dial tcp 10.134.0.32:6567: connect: connection refused"
May 28 11:22:53.910303 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:53.909871    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:22:54.843657 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:54.843619    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused"
May 28 11:22:54.843823 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:54.843742    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:22:55.232369 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.232336    2552 generic.go:358] "Generic (PLEG): container finished" podID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerID="4297fb7119c4c428a1910d54239124a9019156f496727074ed32e42fd61f3344" exitCode=137
May 28 11:22:55.232776 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.232401    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" event={"ID":"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d","Type":"ContainerDied","Data":"4297fb7119c4c428a1910d54239124a9019156f496727074ed32e42fd61f3344"}
May 28 11:22:55.234581 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.234550    2552 generic.go:358] "Generic (PLEG): container finished" podID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerID="f7669d141460c23ca3ac356e05a0464ba94e1a2d1eb18e3241689cf0995729cd" exitCode=137
May 28 11:22:55.234698 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.234603    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" event={"ID":"0da1cf97-4702-4271-b2ca-edb6bdad80d7","Type":"ContainerDied","Data":"f7669d141460c23ca3ac356e05a0464ba94e1a2d1eb18e3241689cf0995729cd"}
May 28 11:22:55.491284 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.491231    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:22:55.494283 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.494258    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:22:55.604518 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.604484    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0da1cf97-4702-4271-b2ca-edb6bdad80d7-registry-tls\") pod \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") "
May 28 11:22:55.604824 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.604538    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-online-tls\") pod \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") "
May 28 11:22:55.604824 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.604560    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjkpp\" (UniqueName: \"kubernetes.io/projected/0da1cf97-4702-4271-b2ca-edb6bdad80d7-kube-api-access-bjkpp\") pod \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") "
May 28 11:22:55.604824 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.604581    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pm5m\" (UniqueName: \"kubernetes.io/projected/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-kube-api-access-5pm5m\") pod \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") "
May 28 11:22:55.604824 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.604780    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0da1cf97-4702-4271-b2ca-edb6bdad80d7-feast-data\") pod \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") "
May 28 11:22:55.605033 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.604849    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-feast-data\") pod \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") "
May 28 11:22:55.605033 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.604932    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0da1cf97-4702-4271-b2ca-edb6bdad80d7-online-tls\") pod \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\" (UID: \"0da1cf97-4702-4271-b2ca-edb6bdad80d7\") "
May 28 11:22:55.605033 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.604987    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-registry-tls\") pod \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\" (UID: \"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d\") "
May 28 11:22:55.605374 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.605325    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da1cf97-4702-4271-b2ca-edb6bdad80d7-feast-data" (OuterVolumeSpecName: "feast-data") pod "0da1cf97-4702-4271-b2ca-edb6bdad80d7" (UID: "0da1cf97-4702-4271-b2ca-edb6bdad80d7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 11:22:55.606860 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.606831    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-kube-api-access-5pm5m" (OuterVolumeSpecName: "kube-api-access-5pm5m") pod "b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" (UID: "b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d"). InnerVolumeSpecName "kube-api-access-5pm5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:22:55.607174 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.607152    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da1cf97-4702-4271-b2ca-edb6bdad80d7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0da1cf97-4702-4271-b2ca-edb6bdad80d7" (UID: "0da1cf97-4702-4271-b2ca-edb6bdad80d7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:22:55.607233 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.607203    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-online-tls" (OuterVolumeSpecName: "online-tls") pod "b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" (UID: "b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:22:55.607408 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.607381    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da1cf97-4702-4271-b2ca-edb6bdad80d7-online-tls" (OuterVolumeSpecName: "online-tls") pod "0da1cf97-4702-4271-b2ca-edb6bdad80d7" (UID: "0da1cf97-4702-4271-b2ca-edb6bdad80d7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:22:55.607512 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.607432    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" (UID: "b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 11:22:55.607512 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.607476    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da1cf97-4702-4271-b2ca-edb6bdad80d7-kube-api-access-bjkpp" (OuterVolumeSpecName: "kube-api-access-bjkpp") pod "0da1cf97-4702-4271-b2ca-edb6bdad80d7" (UID: "0da1cf97-4702-4271-b2ca-edb6bdad80d7"). InnerVolumeSpecName "kube-api-access-bjkpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:22:55.614796 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.614774    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-feast-data" (OuterVolumeSpecName: "feast-data") pod "b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" (UID: "b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 11:22:55.706510 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.706465    2552 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0da1cf97-4702-4271-b2ca-edb6bdad80d7-feast-data\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:22:55.706510 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.706496    2552 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-feast-data\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:22:55.706510 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.706507    2552 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0da1cf97-4702-4271-b2ca-edb6bdad80d7-online-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:22:55.706510 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.706515    2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-registry-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:22:55.706760 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.706523    2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0da1cf97-4702-4271-b2ca-edb6bdad80d7-registry-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:22:55.706760 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.706531    2552 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-online-tls\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:22:55.706760 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.706540    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjkpp\" (UniqueName: \"kubernetes.io/projected/0da1cf97-4702-4271-b2ca-edb6bdad80d7-kube-api-access-bjkpp\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:22:55.706760 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:55.706550    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pm5m\" (UniqueName: \"kubernetes.io/projected/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d-kube-api-access-5pm5m\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:22:56.240043 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.240015    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"
May 28 11:22:56.240539 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.240010    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz" event={"ID":"0da1cf97-4702-4271-b2ca-edb6bdad80d7","Type":"ContainerDied","Data":"feeca8e720a209f3fa54f774729f2db920c95c84de045458fa173e45be396237"}
May 28 11:22:56.240539 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.240160    2552 scope.go:117] "RemoveContainer" containerID="a876019623ac647c7ea2bf99e6c211fae304bf86e70cf33830352f7c0111b333"
May 28 11:22:56.242274 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.242246    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"
May 28 11:22:56.242274 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.242254    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw" event={"ID":"b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d","Type":"ContainerDied","Data":"d851ba3120dca516e0c6798de4b36e29b19f4117c742565e3ace49281af27516"}
May 28 11:22:56.248779 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.248763    2552 scope.go:117] "RemoveContainer" containerID="f7669d141460c23ca3ac356e05a0464ba94e1a2d1eb18e3241689cf0995729cd"
May 28 11:22:56.257185 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.257170    2552 scope.go:117] "RemoveContainer" containerID="4fece75cc421c78cf1bf5e7ff24d293a9cd019b815d4a59a6139798b98c80955"
May 28 11:22:56.263846 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.263825    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"]
May 28 11:22:56.264820 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.264791    2552 scope.go:117] "RemoveContainer" containerID="bc5bdf494adac3e20c016114ac6511c4af909719f273155986c762c12b3c4675"
May 28 11:22:56.267760 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.267741    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-bfdb7cf9b-q86zz"]
May 28 11:22:56.271908 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.271889    2552 scope.go:117] "RemoveContainer" containerID="484c7238d7ba0f929ccfff12b52ab81192187bb684fbea3beee86f9001c42094"
May 28 11:22:56.278708 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.278687    2552 scope.go:117] "RemoveContainer" containerID="4297fb7119c4c428a1910d54239124a9019156f496727074ed32e42fd61f3344"
May 28 11:22:56.280197 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.280176    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"]
May 28 11:22:56.283955 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.283935    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-844579f4c-kz5lw"]
May 28 11:22:56.286492 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.286476    2552 scope.go:117] "RemoveContainer" containerID="27ea1576395299c1ff4c3d964fecc4af32dfead149680fac877c1cb6014585a2"
May 28 11:22:56.294679 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.294663    2552 scope.go:117] "RemoveContainer" containerID="f3dd4bcd98380ff66bf2b6be7dc516e778c9bb4ef7594fca58b1ecb6e291f646"
May 28 11:22:56.517086 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.517010    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" path="/var/lib/kubelet/pods/0da1cf97-4702-4271-b2ca-edb6bdad80d7/volumes"
May 28 11:22:56.517557 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:22:56.517542    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" path="/var/lib/kubelet/pods/b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d/volumes"
May 28 11:23:04.483776 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:23:04.483748    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:23:04.490760 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:23:04.490737    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:28:04.504080 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:28:04.504055    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:28:04.510930 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:28:04.510870    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:33:04.523805 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:04.523735    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:33:04.531644 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:04.531621    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:33:18.596275 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596240    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xkgl6/must-gather-k8zn6"]
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596532    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="feast-init"
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596543    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="feast-init"
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596556    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="online"
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596562    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="online"
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596575    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="registry"
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596580    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="registry"
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596588    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="feast-apply"
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596593    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="feast-apply"
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596600    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="feast-init"
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596605    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="feast-init"
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596613    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="online"
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596618    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="online"
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596625    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="registry"
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596630    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="registry"
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596640    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b875cbdc-d394-437a-85bd-a4ee017d217a" containerName="postgres"
May 28 11:33:18.596640 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596645    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b875cbdc-d394-437a-85bd-a4ee017d217a" containerName="postgres"
May 28 11:33:18.597192 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596650    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="feast-apply"
May 28 11:33:18.597192 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596656    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="feast-apply"
May 28 11:33:18.597192 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596698    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="registry"
May 28 11:33:18.597192 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596709    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="registry"
May 28 11:33:18.597192 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596715    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="b875cbdc-d394-437a-85bd-a4ee017d217a" containerName="postgres"
May 28 11:33:18.597192 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596721    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="b6c4fd09-3cf0-4f96-8abd-20426b8b0f0d" containerName="online"
May 28 11:33:18.597192 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.596728    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="0da1cf97-4702-4271-b2ca-edb6bdad80d7" containerName="online"
May 28 11:33:18.599621 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.599601    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xkgl6/must-gather-k8zn6"
May 28 11:33:18.601719 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.601699    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xkgl6\"/\"default-dockercfg-7xfpk\""
May 28 11:33:18.601809 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.601699    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xkgl6\"/\"kube-root-ca.crt\""
May 28 11:33:18.602242 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.602223    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xkgl6\"/\"openshift-service-ca.crt\""
May 28 11:33:18.608537 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.608513    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xkgl6/must-gather-k8zn6"]
May 28 11:33:18.676407 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.676384    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d2b48184-9d83-4c1c-af1f-6c70d25ac2df-must-gather-output\") pod \"must-gather-k8zn6\" (UID: \"d2b48184-9d83-4c1c-af1f-6c70d25ac2df\") " pod="openshift-must-gather-xkgl6/must-gather-k8zn6"
May 28 11:33:18.676519 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.676424    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klkf6\" (UniqueName: \"kubernetes.io/projected/d2b48184-9d83-4c1c-af1f-6c70d25ac2df-kube-api-access-klkf6\") pod \"must-gather-k8zn6\" (UID: \"d2b48184-9d83-4c1c-af1f-6c70d25ac2df\") " pod="openshift-must-gather-xkgl6/must-gather-k8zn6"
May 28 11:33:18.777223 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.777201    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d2b48184-9d83-4c1c-af1f-6c70d25ac2df-must-gather-output\") pod \"must-gather-k8zn6\" (UID: \"d2b48184-9d83-4c1c-af1f-6c70d25ac2df\") " pod="openshift-must-gather-xkgl6/must-gather-k8zn6"
May 28 11:33:18.777315 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.777239    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-klkf6\" (UniqueName: \"kubernetes.io/projected/d2b48184-9d83-4c1c-af1f-6c70d25ac2df-kube-api-access-klkf6\") pod \"must-gather-k8zn6\" (UID: \"d2b48184-9d83-4c1c-af1f-6c70d25ac2df\") " pod="openshift-must-gather-xkgl6/must-gather-k8zn6"
May 28 11:33:18.777508 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.777491    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d2b48184-9d83-4c1c-af1f-6c70d25ac2df-must-gather-output\") pod \"must-gather-k8zn6\" (UID: \"d2b48184-9d83-4c1c-af1f-6c70d25ac2df\") " pod="openshift-must-gather-xkgl6/must-gather-k8zn6"
May 28 11:33:18.786572 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.786549    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klkf6\" (UniqueName: \"kubernetes.io/projected/d2b48184-9d83-4c1c-af1f-6c70d25ac2df-kube-api-access-klkf6\") pod \"must-gather-k8zn6\" (UID: \"d2b48184-9d83-4c1c-af1f-6c70d25ac2df\") " pod="openshift-must-gather-xkgl6/must-gather-k8zn6"
May 28 11:33:18.909113 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:18.909027    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xkgl6/must-gather-k8zn6"
May 28 11:33:19.028542 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:19.028518    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xkgl6/must-gather-k8zn6"]
May 28 11:33:19.031045 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:33:19.031012    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2b48184_9d83_4c1c_af1f_6c70d25ac2df.slice/crio-3d33f4557c2fe71ba8ed6fea15fedeec137a7d3b8cb713efdf03ce0dc5a48355 WatchSource:0}: Error finding container 3d33f4557c2fe71ba8ed6fea15fedeec137a7d3b8cb713efdf03ce0dc5a48355: Status 404 returned error can't find the container with id 3d33f4557c2fe71ba8ed6fea15fedeec137a7d3b8cb713efdf03ce0dc5a48355
May 28 11:33:19.032734 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:19.032716    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 11:33:19.985325 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:19.985283    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xkgl6/must-gather-k8zn6" event={"ID":"d2b48184-9d83-4c1c-af1f-6c70d25ac2df","Type":"ContainerStarted","Data":"3d33f4557c2fe71ba8ed6fea15fedeec137a7d3b8cb713efdf03ce0dc5a48355"}
May 28 11:33:26.009450 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:26.009409    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xkgl6/must-gather-k8zn6" event={"ID":"d2b48184-9d83-4c1c-af1f-6c70d25ac2df","Type":"ContainerStarted","Data":"7ce7f8a95ab656b38a4544d6b06f5cc48dc653652f99f992a90bdb2e630646dc"}
May 28 11:33:26.009450 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:26.009455    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xkgl6/must-gather-k8zn6" event={"ID":"d2b48184-9d83-4c1c-af1f-6c70d25ac2df","Type":"ContainerStarted","Data":"abf19d2adef001ae11f498f09991b2b493409bd89dc5b7203054e0a03fa5cd31"}
May 28 11:33:26.024683 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:26.024633    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-xkgl6/must-gather-k8zn6" podStartSLOduration=1.980635702 podStartE2EDuration="8.024616151s" podCreationTimestamp="2026-05-28 11:33:18 +0000 UTC" firstStartedPulling="2026-05-28 11:33:19.032841365 +0000 UTC m=+1815.071938322" lastFinishedPulling="2026-05-28 11:33:25.076821811 +0000 UTC m=+1821.115918771" observedRunningTime="2026-05-28 11:33:26.023374301 +0000 UTC m=+1822.062471281" watchObservedRunningTime="2026-05-28 11:33:26.024616151 +0000 UTC m=+1822.063713180"
May 28 11:33:35.042898 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:35.042866    2552 generic.go:358] "Generic (PLEG): container finished" podID="d2b48184-9d83-4c1c-af1f-6c70d25ac2df" containerID="abf19d2adef001ae11f498f09991b2b493409bd89dc5b7203054e0a03fa5cd31" exitCode=0
May 28 11:33:35.043310 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:35.042939    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xkgl6/must-gather-k8zn6" event={"ID":"d2b48184-9d83-4c1c-af1f-6c70d25ac2df","Type":"ContainerDied","Data":"abf19d2adef001ae11f498f09991b2b493409bd89dc5b7203054e0a03fa5cd31"}
May 28 11:33:35.043310 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:35.043244    2552 scope.go:117] "RemoveContainer" containerID="abf19d2adef001ae11f498f09991b2b493409bd89dc5b7203054e0a03fa5cd31"
May 28 11:33:35.637112 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:35.637079    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-xkgl6_must-gather-k8zn6_d2b48184-9d83-4c1c-af1f-6c70d25ac2df/gather/0.log"
May 28 11:33:38.641924 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:38.641894    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4znlc_4e5305c2-8214-4b76-9e9b-56fd284dbe37/global-pull-secret-syncer/0.log"
May 28 11:33:38.784835 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:38.784805    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8xtm7_51fa9d4e-2280-4b26-8d0d-c9649fa19f51/konnectivity-agent/0.log"
May 28 11:33:38.885659 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:38.885627    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-107.ec2.internal_3a5e6230d598a6959a877daf9ec83d5e/haproxy/0.log"
May 28 11:33:40.988970 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:40.988932    2552 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-xkgl6/must-gather-k8zn6"]
May 28 11:33:40.989566 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:40.989234    2552 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-xkgl6/must-gather-k8zn6" podUID="d2b48184-9d83-4c1c-af1f-6c70d25ac2df" containerName="copy" containerID="cri-o://7ce7f8a95ab656b38a4544d6b06f5cc48dc653652f99f992a90bdb2e630646dc" gracePeriod=2
May 28 11:33:40.996503 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:40.996467    2552 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-xkgl6/must-gather-k8zn6"]
May 28 11:33:41.216378 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:41.216354    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-xkgl6_must-gather-k8zn6_d2b48184-9d83-4c1c-af1f-6c70d25ac2df/copy/0.log"
May 28 11:33:41.216704 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:41.216686    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xkgl6/must-gather-k8zn6"
May 28 11:33:41.218380 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:41.218350    2552 status_manager.go:895] "Failed to get status for pod" podUID="d2b48184-9d83-4c1c-af1f-6c70d25ac2df" pod="openshift-must-gather-xkgl6/must-gather-k8zn6" err="pods \"must-gather-k8zn6\" is forbidden: User \"system:node:ip-10-0-133-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-xkgl6\": no relationship found between node 'ip-10-0-133-107.ec2.internal' and this object"
May 28 11:33:41.356219 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:41.356147    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klkf6\" (UniqueName: \"kubernetes.io/projected/d2b48184-9d83-4c1c-af1f-6c70d25ac2df-kube-api-access-klkf6\") pod \"d2b48184-9d83-4c1c-af1f-6c70d25ac2df\" (UID: \"d2b48184-9d83-4c1c-af1f-6c70d25ac2df\") "
May 28 11:33:41.356340 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:41.356220    2552 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d2b48184-9d83-4c1c-af1f-6c70d25ac2df-must-gather-output\") pod \"d2b48184-9d83-4c1c-af1f-6c70d25ac2df\" (UID: \"d2b48184-9d83-4c1c-af1f-6c70d25ac2df\") "
May 28 11:33:41.356592 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:41.356571    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b48184-9d83-4c1c-af1f-6c70d25ac2df-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "d2b48184-9d83-4c1c-af1f-6c70d25ac2df" (UID: "d2b48184-9d83-4c1c-af1f-6c70d25ac2df"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 11:33:41.358268 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:41.358248    2552 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b48184-9d83-4c1c-af1f-6c70d25ac2df-kube-api-access-klkf6" (OuterVolumeSpecName: "kube-api-access-klkf6") pod "d2b48184-9d83-4c1c-af1f-6c70d25ac2df" (UID: "d2b48184-9d83-4c1c-af1f-6c70d25ac2df"). InnerVolumeSpecName "kube-api-access-klkf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 11:33:41.457073 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:41.457051    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klkf6\" (UniqueName: \"kubernetes.io/projected/d2b48184-9d83-4c1c-af1f-6c70d25ac2df-kube-api-access-klkf6\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:33:41.457073 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:41.457071    2552 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d2b48184-9d83-4c1c-af1f-6c70d25ac2df-must-gather-output\") on node \"ip-10-0-133-107.ec2.internal\" DevicePath \"\""
May 28 11:33:41.966359 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:41.966328    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b57bf21e-ab6e-496e-b9ea-5589436e0ed9/alertmanager/0.log"
May 28 11:33:41.995090 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:41.995068    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b57bf21e-ab6e-496e-b9ea-5589436e0ed9/config-reloader/0.log"
May 28 11:33:42.022830 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.022808    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b57bf21e-ab6e-496e-b9ea-5589436e0ed9/kube-rbac-proxy-web/0.log"
May 28 11:33:42.043393 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.043375    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b57bf21e-ab6e-496e-b9ea-5589436e0ed9/kube-rbac-proxy/0.log"
May 28 11:33:42.062075 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.062054    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-xkgl6_must-gather-k8zn6_d2b48184-9d83-4c1c-af1f-6c70d25ac2df/copy/0.log"
May 28 11:33:42.062396 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.062373    2552 generic.go:358] "Generic (PLEG): container finished" podID="d2b48184-9d83-4c1c-af1f-6c70d25ac2df" containerID="7ce7f8a95ab656b38a4544d6b06f5cc48dc653652f99f992a90bdb2e630646dc" exitCode=143
May 28 11:33:42.062491 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.062423    2552 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xkgl6/must-gather-k8zn6"
May 28 11:33:42.062491 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.062453    2552 scope.go:117] "RemoveContainer" containerID="7ce7f8a95ab656b38a4544d6b06f5cc48dc653652f99f992a90bdb2e630646dc"
May 28 11:33:42.065777 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.065750    2552 status_manager.go:895] "Failed to get status for pod" podUID="d2b48184-9d83-4c1c-af1f-6c70d25ac2df" pod="openshift-must-gather-xkgl6/must-gather-k8zn6" err="pods \"must-gather-k8zn6\" is forbidden: User \"system:node:ip-10-0-133-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-xkgl6\": no relationship found between node 'ip-10-0-133-107.ec2.internal' and this object"
May 28 11:33:42.068788 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.068772    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b57bf21e-ab6e-496e-b9ea-5589436e0ed9/kube-rbac-proxy-metric/0.log"
May 28 11:33:42.070247 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.070226    2552 scope.go:117] "RemoveContainer" containerID="abf19d2adef001ae11f498f09991b2b493409bd89dc5b7203054e0a03fa5cd31"
May 28 11:33:42.072998 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.072974    2552 status_manager.go:895] "Failed to get status for pod" podUID="d2b48184-9d83-4c1c-af1f-6c70d25ac2df" pod="openshift-must-gather-xkgl6/must-gather-k8zn6" err="pods \"must-gather-k8zn6\" is forbidden: User \"system:node:ip-10-0-133-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-xkgl6\": no relationship found between node 'ip-10-0-133-107.ec2.internal' and this object"
May 28 11:33:42.081489 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.081473    2552 scope.go:117] "RemoveContainer" containerID="7ce7f8a95ab656b38a4544d6b06f5cc48dc653652f99f992a90bdb2e630646dc"
May 28 11:33:42.081741 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:33:42.081721    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ce7f8a95ab656b38a4544d6b06f5cc48dc653652f99f992a90bdb2e630646dc\": container with ID starting with 7ce7f8a95ab656b38a4544d6b06f5cc48dc653652f99f992a90bdb2e630646dc not found: ID does not exist" containerID="7ce7f8a95ab656b38a4544d6b06f5cc48dc653652f99f992a90bdb2e630646dc"
May 28 11:33:42.081805 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.081747    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ce7f8a95ab656b38a4544d6b06f5cc48dc653652f99f992a90bdb2e630646dc"} err="failed to get container status \"7ce7f8a95ab656b38a4544d6b06f5cc48dc653652f99f992a90bdb2e630646dc\": rpc error: code = NotFound desc = could not find container \"7ce7f8a95ab656b38a4544d6b06f5cc48dc653652f99f992a90bdb2e630646dc\": container with ID starting with 7ce7f8a95ab656b38a4544d6b06f5cc48dc653652f99f992a90bdb2e630646dc not found: ID does not exist"
May 28 11:33:42.081805 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.081764    2552 scope.go:117] "RemoveContainer" containerID="abf19d2adef001ae11f498f09991b2b493409bd89dc5b7203054e0a03fa5cd31"
May 28 11:33:42.082027 ip-10-0-133-107 kubenswrapper[2552]: E0528 11:33:42.081990    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf19d2adef001ae11f498f09991b2b493409bd89dc5b7203054e0a03fa5cd31\": container with ID starting with abf19d2adef001ae11f498f09991b2b493409bd89dc5b7203054e0a03fa5cd31 not found: ID does not exist" containerID="abf19d2adef001ae11f498f09991b2b493409bd89dc5b7203054e0a03fa5cd31"
May 28 11:33:42.082027 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.082011    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf19d2adef001ae11f498f09991b2b493409bd89dc5b7203054e0a03fa5cd31"} err="failed to get container status \"abf19d2adef001ae11f498f09991b2b493409bd89dc5b7203054e0a03fa5cd31\": rpc error: code = NotFound desc = could not find container \"abf19d2adef001ae11f498f09991b2b493409bd89dc5b7203054e0a03fa5cd31\": container with ID starting with abf19d2adef001ae11f498f09991b2b493409bd89dc5b7203054e0a03fa5cd31 not found: ID does not exist"
May 28 11:33:42.093227 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.093213    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b57bf21e-ab6e-496e-b9ea-5589436e0ed9/prom-label-proxy/0.log"
May 28 11:33:42.119118 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.119103    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b57bf21e-ab6e-496e-b9ea-5589436e0ed9/init-config-reloader/0.log"
May 28 11:33:42.272716 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.272689    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-ps5wd_1d04a73f-8af3-447c-b32f-2d22e86dac3a/monitoring-plugin/0.log"
May 28 11:33:42.469589 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.469561    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-clc5r_1987da86-b332-4655-b7d5-5c1061fd900a/node-exporter/0.log"
May 28 11:33:42.492277 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.492250    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-clc5r_1987da86-b332-4655-b7d5-5c1061fd900a/kube-rbac-proxy/0.log"
May 28 11:33:42.516603 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.516583    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-clc5r_1987da86-b332-4655-b7d5-5c1061fd900a/init-textfile/0.log"
May 28 11:33:42.517003 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.516984    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2b48184-9d83-4c1c-af1f-6c70d25ac2df" path="/var/lib/kubelet/pods/d2b48184-9d83-4c1c-af1f-6c70d25ac2df/volumes"
May 28 11:33:42.817251 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.817223    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-mf2pp_37406e38-ff29-456b-bc48-581238e8e9e1/prometheus-operator/0.log"
May 28 11:33:42.837532 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.837507    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-mf2pp_37406e38-ff29-456b-bc48-581238e8e9e1/kube-rbac-proxy/0.log"
May 28 11:33:42.898818 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.898799    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85cfd9cc64-kk8pm_8179a300-6b5b-428e-aa0b-c0e4beafd3d5/telemeter-client/0.log"
May 28 11:33:42.925366 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.925349    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85cfd9cc64-kk8pm_8179a300-6b5b-428e-aa0b-c0e4beafd3d5/reload/0.log"
May 28 11:33:42.947320 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:42.947304    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85cfd9cc64-kk8pm_8179a300-6b5b-428e-aa0b-c0e4beafd3d5/kube-rbac-proxy/0.log"
May 28 11:33:44.593554 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:44.593481    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/2.log"
May 28 11:33:44.597608 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:44.597589    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-s5n2k_99a5f73b-ab15-4705-aa51-6f19d1e0f558/console-operator/3.log"
May 28 11:33:44.946591 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:44.946520    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-llgn8_150584db-a4a1-496c-b65a-33ca9280dd10/download-server/0.log"
May 28 11:33:45.175785 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.175749    2552 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"]
May 28 11:33:45.176154 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.176119    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b48184-9d83-4c1c-af1f-6c70d25ac2df" containerName="copy"
May 28 11:33:45.176154 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.176155    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b48184-9d83-4c1c-af1f-6c70d25ac2df" containerName="copy"
May 28 11:33:45.176309 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.176174    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b48184-9d83-4c1c-af1f-6c70d25ac2df" containerName="gather"
May 28 11:33:45.176309 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.176182    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b48184-9d83-4c1c-af1f-6c70d25ac2df" containerName="gather"
May 28 11:33:45.176309 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.176264    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="d2b48184-9d83-4c1c-af1f-6c70d25ac2df" containerName="gather"
May 28 11:33:45.176309 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.176284    2552 memory_manager.go:356] "RemoveStaleState removing state" podUID="d2b48184-9d83-4c1c-af1f-6c70d25ac2df" containerName="copy"
May 28 11:33:45.179421 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.179400    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.181264 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.181246    2552 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f968n\"/\"default-dockercfg-vc2f4\""
May 28 11:33:45.181769 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.181749    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f968n\"/\"kube-root-ca.crt\""
May 28 11:33:45.181836 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.181780    2552 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f968n\"/\"openshift-service-ca.crt\""
May 28 11:33:45.186426 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.186407    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"]
May 28 11:33:45.284924 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.284900    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2999894d-a925-4f78-b28e-94e5a0b08cb9-sys\") pod \"perf-node-gather-daemonset-skfbh\" (UID: \"2999894d-a925-4f78-b28e-94e5a0b08cb9\") " pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.285057 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.284928    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2999894d-a925-4f78-b28e-94e5a0b08cb9-proc\") pod \"perf-node-gather-daemonset-skfbh\" (UID: \"2999894d-a925-4f78-b28e-94e5a0b08cb9\") " pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.285057 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.284950    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkqv6\" (UniqueName: \"kubernetes.io/projected/2999894d-a925-4f78-b28e-94e5a0b08cb9-kube-api-access-xkqv6\") pod \"perf-node-gather-daemonset-skfbh\" (UID: \"2999894d-a925-4f78-b28e-94e5a0b08cb9\") " pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.285057 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.284989    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2999894d-a925-4f78-b28e-94e5a0b08cb9-podres\") pod \"perf-node-gather-daemonset-skfbh\" (UID: \"2999894d-a925-4f78-b28e-94e5a0b08cb9\") " pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.285057 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.285034    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2999894d-a925-4f78-b28e-94e5a0b08cb9-lib-modules\") pod \"perf-node-gather-daemonset-skfbh\" (UID: \"2999894d-a925-4f78-b28e-94e5a0b08cb9\") " pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.386297 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.386274    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2999894d-a925-4f78-b28e-94e5a0b08cb9-podres\") pod \"perf-node-gather-daemonset-skfbh\" (UID: \"2999894d-a925-4f78-b28e-94e5a0b08cb9\") " pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.386402 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.386307    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2999894d-a925-4f78-b28e-94e5a0b08cb9-lib-modules\") pod \"perf-node-gather-daemonset-skfbh\" (UID: \"2999894d-a925-4f78-b28e-94e5a0b08cb9\") " pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.386402 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.386357    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2999894d-a925-4f78-b28e-94e5a0b08cb9-sys\") pod \"perf-node-gather-daemonset-skfbh\" (UID: \"2999894d-a925-4f78-b28e-94e5a0b08cb9\") " pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.386402 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.386373    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2999894d-a925-4f78-b28e-94e5a0b08cb9-proc\") pod \"perf-node-gather-daemonset-skfbh\" (UID: \"2999894d-a925-4f78-b28e-94e5a0b08cb9\") " pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.386402 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.386390    2552 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xkqv6\" (UniqueName: \"kubernetes.io/projected/2999894d-a925-4f78-b28e-94e5a0b08cb9-kube-api-access-xkqv6\") pod \"perf-node-gather-daemonset-skfbh\" (UID: \"2999894d-a925-4f78-b28e-94e5a0b08cb9\") " pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.386546 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.386426    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2999894d-a925-4f78-b28e-94e5a0b08cb9-podres\") pod \"perf-node-gather-daemonset-skfbh\" (UID: \"2999894d-a925-4f78-b28e-94e5a0b08cb9\") " pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.386546 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.386455    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2999894d-a925-4f78-b28e-94e5a0b08cb9-sys\") pod \"perf-node-gather-daemonset-skfbh\" (UID: \"2999894d-a925-4f78-b28e-94e5a0b08cb9\") " pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.386546 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.386474    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2999894d-a925-4f78-b28e-94e5a0b08cb9-proc\") pod \"perf-node-gather-daemonset-skfbh\" (UID: \"2999894d-a925-4f78-b28e-94e5a0b08cb9\") " pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.386546 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.386534    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2999894d-a925-4f78-b28e-94e5a0b08cb9-lib-modules\") pod \"perf-node-gather-daemonset-skfbh\" (UID: \"2999894d-a925-4f78-b28e-94e5a0b08cb9\") " pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.388017 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.387999    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-5lgwk_3aa30366-ea47-4dad-ab45-38932e616d71/volume-data-source-validator/0.log"
May 28 11:33:45.393698 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.393677    2552 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkqv6\" (UniqueName: \"kubernetes.io/projected/2999894d-a925-4f78-b28e-94e5a0b08cb9-kube-api-access-xkqv6\") pod \"perf-node-gather-daemonset-skfbh\" (UID: \"2999894d-a925-4f78-b28e-94e5a0b08cb9\") " pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.489950 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.489929    2552 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:45.613421 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:45.613392    2552 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"]
May 28 11:33:45.616688 ip-10-0-133-107 kubenswrapper[2552]: W0528 11:33:45.616652    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2999894d_a925_4f78_b28e_94e5a0b08cb9.slice/crio-d9d6a4f0607c9be248cdab1e34329f0b71c55717e89a1cc79f7d91edbbcd5ae5 WatchSource:0}: Error finding container d9d6a4f0607c9be248cdab1e34329f0b71c55717e89a1cc79f7d91edbbcd5ae5: Status 404 returned error can't find the container with id d9d6a4f0607c9be248cdab1e34329f0b71c55717e89a1cc79f7d91edbbcd5ae5
May 28 11:33:46.074932 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:46.074899    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh" event={"ID":"2999894d-a925-4f78-b28e-94e5a0b08cb9","Type":"ContainerStarted","Data":"ff500ebaa3aa21ae215497f6d407133b0e6f98b5eba1a59784c18830dfb6dc16"}
May 28 11:33:46.075116 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:46.074936    2552 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh" event={"ID":"2999894d-a925-4f78-b28e-94e5a0b08cb9","Type":"ContainerStarted","Data":"d9d6a4f0607c9be248cdab1e34329f0b71c55717e89a1cc79f7d91edbbcd5ae5"}
May 28 11:33:46.075116 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:46.074970    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:46.083834 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:46.083810    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7c95j_566c95b3-6db1-4296-8ec0-f823019d6a40/dns/0.log"
May 28 11:33:46.091186 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:46.091116    2552 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh" podStartSLOduration=1.091102291 podStartE2EDuration="1.091102291s" podCreationTimestamp="2026-05-28 11:33:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 11:33:46.090271689 +0000 UTC m=+1842.129368668" watchObservedRunningTime="2026-05-28 11:33:46.091102291 +0000 UTC m=+1842.130199270"
May 28 11:33:46.142689 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:46.142665    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7c95j_566c95b3-6db1-4296-8ec0-f823019d6a40/kube-rbac-proxy/0.log"
May 28 11:33:46.209590 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:46.209571    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4lh8f_cba5e87a-42ea-447e-8c21-439ce0bbb9fd/dns-node-resolver/0.log"
May 28 11:33:46.654655 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:46.654629    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hsvzs_4fd526dd-8e8c-4600-a0cb-182ce3516c6f/node-ca/0.log"
May 28 11:33:47.353583 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:47.353554    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c79db4c74-j496b_037f095d-ebe7-44bd-8c85-0902026307ae/router/0.log"
May 28 11:33:47.650754 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:47.650681    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8pg7p_99ba1953-8c33-4ce7-81dd-ff524369264f/serve-healthcheck-canary/0.log"
May 28 11:33:48.086980 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:48.086954    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-whkx6_1a64635e-7734-4e92-8253-2de6548df3c2/kube-rbac-proxy/0.log"
May 28 11:33:48.107030 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:48.107006    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-whkx6_1a64635e-7734-4e92-8253-2de6548df3c2/exporter/0.log"
May 28 11:33:48.127555 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:48.127532    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-whkx6_1a64635e-7734-4e92-8253-2de6548df3c2/extractor/0.log"
May 28 11:33:52.087300 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:52.087276    2552 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f968n/perf-node-gather-daemonset-skfbh"
May 28 11:33:52.264089 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:52.264059    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-vvkxt_c4865c2f-d302-411e-ac37-42d66f527743/kube-storage-version-migrator-operator/1.log"
May 28 11:33:52.264886 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:52.264870    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-vvkxt_c4865c2f-d302-411e-ac37-42d66f527743/kube-storage-version-migrator-operator/0.log"
May 28 11:33:53.237294 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:53.237269    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j8tns_e78bfc6f-02aa-4a5c-afc7-d3a907840f53/kube-multus-additional-cni-plugins/0.log"
May 28 11:33:53.259312 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:53.259289    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j8tns_e78bfc6f-02aa-4a5c-afc7-d3a907840f53/egress-router-binary-copy/0.log"
May 28 11:33:53.282459 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:53.282440    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j8tns_e78bfc6f-02aa-4a5c-afc7-d3a907840f53/cni-plugins/0.log"
May 28 11:33:53.302977 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:53.302958    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j8tns_e78bfc6f-02aa-4a5c-afc7-d3a907840f53/bond-cni-plugin/0.log"
May 28 11:33:53.324000 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:53.323983    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j8tns_e78bfc6f-02aa-4a5c-afc7-d3a907840f53/routeoverride-cni/0.log"
May 28 11:33:53.345974 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:53.345952    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j8tns_e78bfc6f-02aa-4a5c-afc7-d3a907840f53/whereabouts-cni-bincopy/0.log"
May 28 11:33:53.366461 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:53.366440    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j8tns_e78bfc6f-02aa-4a5c-afc7-d3a907840f53/whereabouts-cni/0.log"
May 28 11:33:53.584864 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:53.584801    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-tprqr_0d2c92fb-bc6d-4c6e-b59a-d9c745d08344/kube-multus/0.log"
May 28 11:33:53.605098 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:53.605081    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5jcnw_1f58f4ea-58e7-4a59-afd8-b41076bfa130/network-metrics-daemon/0.log"
May 28 11:33:53.631093 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:53.631076    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5jcnw_1f58f4ea-58e7-4a59-afd8-b41076bfa130/kube-rbac-proxy/0.log"
May 28 11:33:54.614486 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:54.614411    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhbhc_2c6961c4-e7e5-46bd-aa0d-2dc026df36e1/ovn-controller/0.log"
May 28 11:33:54.643354 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:54.643334    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhbhc_2c6961c4-e7e5-46bd-aa0d-2dc026df36e1/ovn-acl-logging/0.log"
May 28 11:33:54.664200 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:54.664175    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhbhc_2c6961c4-e7e5-46bd-aa0d-2dc026df36e1/kube-rbac-proxy-node/0.log"
May 28 11:33:54.693870 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:54.693850    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhbhc_2c6961c4-e7e5-46bd-aa0d-2dc026df36e1/kube-rbac-proxy-ovn-metrics/0.log"
May 28 11:33:54.717853 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:54.717835    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhbhc_2c6961c4-e7e5-46bd-aa0d-2dc026df36e1/northd/0.log"
May 28 11:33:54.745946 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:54.745930    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhbhc_2c6961c4-e7e5-46bd-aa0d-2dc026df36e1/nbdb/0.log"
May 28 11:33:54.770067 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:54.770047    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhbhc_2c6961c4-e7e5-46bd-aa0d-2dc026df36e1/sbdb/0.log"
May 28 11:33:54.868794 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:54.868731    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fhbhc_2c6961c4-e7e5-46bd-aa0d-2dc026df36e1/ovnkube-controller/0.log"
May 28 11:33:56.379054 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:56.378982    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-kc5jm_c3baf54a-14a3-45e2-89a8-ddc61fd0b8f2/network-check-target-container/0.log"
May 28 11:33:57.266391 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:57.266361    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-s8hjt_7546d43c-0df3-4fa0-9258-908e2c006bb8/iptables-alerter/0.log"
May 28 11:33:57.898299 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:57.898273    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-wzmbz_e9125deb-cfc1-432d-acb8-31f019b7987b/tuned/0.log"
May 28 11:33:59.483021 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:59.482994    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-9fhq4_6bf8f86f-b15e-457d-a25a-9472411ea86d/cluster-samples-operator/0.log"
May 28 11:33:59.505697 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:33:59.505657    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-9fhq4_6bf8f86f-b15e-457d-a25a-9472411ea86d/cluster-samples-operator-watch/0.log"
May 28 11:34:00.673885 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:34:00.673818    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-67d667c7d-hdhh6_c9492ee6-80d4-41e0-8fab-22a8d6039a3a/service-ca-controller/0.log"
May 28 11:34:01.016780 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:34:01.016750    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-66nrg_a3c317fb-48c6-4e83-ad01-06395193c20c/csi-driver/0.log"
May 28 11:34:01.038537 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:34:01.038511    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-66nrg_a3c317fb-48c6-4e83-ad01-06395193c20c/csi-node-driver-registrar/0.log"
May 28 11:34:01.062882 ip-10-0-133-107 kubenswrapper[2552]: I0528 11:34:01.062863    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-66nrg_a3c317fb-48c6-4e83-ad01-06395193c20c/csi-liveness-probe/0.log"