May 28 23:02:06.670153 ip-10-0-143-104 systemd[1]: Starting Kubernetes Kubelet...
May 28 23:02:07.167573 ip-10-0-143-104 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 23:02:07.167573 ip-10-0-143-104 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 28 23:02:07.168082 ip-10-0-143-104 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 23:02:07.168082 ip-10-0-143-104 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 28 23:02:07.168082 ip-10-0-143-104 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 23:02:07.170463 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.170336    2577 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 28 23:02:07.179842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179820    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 23:02:07.179842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179838    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 23:02:07.179842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179843    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 23:02:07.179842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179847    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 23:02:07.179842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179851    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179854    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179858    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179861    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179881    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179885    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179888    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179891    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179894    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179897    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179900    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179903    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179906    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179909    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179911    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179914    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179916    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179919    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179921    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179924    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 23:02:07.180050 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179926    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179937    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179939    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179942    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179945    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179948    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179951    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179953    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179956    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179958    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179961    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179963    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179966    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179968    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179973    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179976    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179978    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179981    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179983    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179986    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 23:02:07.180512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179988    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179991    2577 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179995    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.179999    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180001    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180004    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180006    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180009    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180011    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180014    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180016    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180019    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180021    2577 feature_gate.go:328] unrecognized feature gate: Example
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180024    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180026    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180029    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180032    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180034    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180037    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 23:02:07.181046 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180039    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180042    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180050    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180054    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180057    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180061    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180063    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180067    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180069    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180072    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180074    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180077    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180079    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180082    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180084    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180093    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180096    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180098    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180101    2577 feature_gate.go:328] unrecognized feature gate: Example2
May 28 23:02:07.181501 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180104    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180106    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180108    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180111    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180559    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180565    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180568    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180570    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180573    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180575    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180580    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180583    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180586    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180588    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180591    2577 feature_gate.go:328] unrecognized feature gate: Example2
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180594    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180597    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180599    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180602    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 23:02:07.181963 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180605    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180608    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180611    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180614    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180617    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180619    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180622    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180624    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180627    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180635    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180638    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180640    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180643    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180645    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180648    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180650    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180653    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180655    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180658    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180660    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 23:02:07.182420 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180663    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180666    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180669    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180671    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180674    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180677    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180679    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180682    2577 feature_gate.go:328] unrecognized feature gate: Example
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180684    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180686    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180689    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180691    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180694    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180696    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180699    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180701    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180704    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180707    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180709    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180712    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 23:02:07.182930 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180714    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180716    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180724    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180727    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180729    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180732    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180734    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180736    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180739    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180741    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180744    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180746    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180749    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180751    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180753    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180759    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180762    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180764    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180767    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180769    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 23:02:07.183431 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180773    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180777    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180780    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180783    2577 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180786    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180789    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180792    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180795    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180798    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180801    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.180803    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180897    2577 flags.go:64] FLAG: --address="0.0.0.0"
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180906    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180916    2577 flags.go:64] FLAG: --anonymous-auth="true"
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180921    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180931    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180935    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180939    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180943    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180947    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180950    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 28 23:02:07.183933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180953    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180956    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180959    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180962    2577 flags.go:64] FLAG: --cgroup-root=""
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180965    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180967    2577 flags.go:64] FLAG: --client-ca-file=""
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180972    2577 flags.go:64] FLAG: --cloud-config=""
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180974    2577 flags.go:64] FLAG: --cloud-provider="external"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180977    2577 flags.go:64] FLAG: --cluster-dns="[]"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180984    2577 flags.go:64] FLAG: --cluster-domain=""
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180987    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180990    2577 flags.go:64] FLAG: --config-dir=""
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180992    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180995    2577 flags.go:64] FLAG: --container-log-max-files="5"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.180999    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181002    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181006    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181009    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181012    2577 flags.go:64] FLAG: --contention-profiling="false"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181015    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181018    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181021    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181024    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181028    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181031    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 28 23:02:07.184486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181034    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181037    2577 flags.go:64] FLAG: --enable-load-reader="false"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181045    2577 flags.go:64] FLAG: --enable-server="true"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181048    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181055    2577 flags.go:64] FLAG: --event-burst="100"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181058    2577 flags.go:64] FLAG: --event-qps="50"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181061    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181064    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181067    2577 flags.go:64] FLAG: --eviction-hard=""
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181071    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181074    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181077    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181079    2577 flags.go:64] FLAG: --eviction-soft=""
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181083    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181086    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181089    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181091    2577 flags.go:64] FLAG: --experimental-mounter-path=""
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181094    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181096    2577 flags.go:64] FLAG: --fail-swap-on="true"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181099    2577 flags.go:64] FLAG: --feature-gates=""
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181103    2577 flags.go:64] FLAG: --file-check-frequency="20s"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181106    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181110    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181113    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181116    2577 flags.go:64] FLAG: --healthz-port="10248"
May 28 23:02:07.185105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181119    2577 flags.go:64] FLAG: --help="false"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181122    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-143-104.ec2.internal"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181125    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181128    2577 flags.go:64] FLAG: --http-check-frequency="20s"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181130    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181134    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181137    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181139    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181142    2577 flags.go:64] FLAG: --image-service-endpoint=""
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181145    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181155    2577 flags.go:64] FLAG: --kube-api-burst="100"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181158    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181161    2577 flags.go:64] FLAG: --kube-api-qps="50"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181164    2577 flags.go:64] FLAG: --kube-reserved=""
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181167    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181169    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181172    2577 flags.go:64] FLAG: --kubelet-cgroups=""
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181175    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181178    2577 flags.go:64] FLAG: --lock-file=""
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181180    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181183    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181186    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181191    2577 flags.go:64] FLAG: --log-json-split-stream="false"
May 28 23:02:07.185698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181194    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181197    2577 flags.go:64] FLAG: --log-text-split-stream="false"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181199    2577 flags.go:64] FLAG: --logging-format="text"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181202    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181205    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181208    2577 flags.go:64] FLAG: --manifest-url=""
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181210    2577 flags.go:64] FLAG: --manifest-url-header=""
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181215    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181218    2577 flags.go:64] FLAG: --max-open-files="1000000"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181222    2577 flags.go:64] FLAG: --max-pods="110"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181225    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181228    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181231    2577 flags.go:64] FLAG: --memory-manager-policy="None"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181233    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181236    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181239    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181242    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181251    2577 flags.go:64] FLAG: --node-status-max-images="50"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181253    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181256    2577 flags.go:64] FLAG: --oom-score-adj="-999"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181265    2577 flags.go:64] FLAG: --pod-cidr=""
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181268    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181273    2577 flags.go:64] FLAG: --pod-manifest-path=""
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181276    2577 flags.go:64] FLAG: --pod-max-pids="-1"
May 28 23:02:07.186266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181279    2577 flags.go:64] FLAG: --pods-per-core="0"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181282    2577 flags.go:64] FLAG: --port="10250"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181285    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181288    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a8e9c73fb1ec8246"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181290    2577 flags.go:64] FLAG: --qos-reserved=""
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181293    2577 flags.go:64] FLAG: --read-only-port="10255"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181296    2577 flags.go:64] FLAG: --register-node="true"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181299    2577 flags.go:64] FLAG: --register-schedulable="true"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181302    2577 flags.go:64] FLAG: --register-with-taints=""
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181305    2577 flags.go:64] FLAG: --registry-burst="10"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181308    2577 flags.go:64] FLAG: --registry-qps="5"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181311    2577 flags.go:64] FLAG: --reserved-cpus=""
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181313    2577 flags.go:64] FLAG: --reserved-memory=""
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181317    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181322    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181325    2577 flags.go:64] FLAG: --rotate-certificates="false"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181328    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181330    2577 flags.go:64] FLAG: --runonce="false"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181333    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181336    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181338    2577 flags.go:64] FLAG: --seccomp-default="false"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181341    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181344    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181347    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181350    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181353    2577 flags.go:64] FLAG: --storage-driver-password="root"
May 28 23:02:07.186831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181356    2577 flags.go:64] FLAG: --storage-driver-secure="false"
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181358    2577 flags.go:64] FLAG: --storage-driver-table="stats"
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181361    2577 flags.go:64] FLAG: --storage-driver-user="root"
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181369    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181373    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181375    2577 flags.go:64] FLAG: --system-cgroups=""
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181378    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181383    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181386    2577 flags.go:64] FLAG: --tls-cert-file=""
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181389    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181395    2577 flags.go:64] FLAG: --tls-min-version=""
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181398    2577 flags.go:64] FLAG: --tls-private-key-file=""
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181401    2577 flags.go:64] FLAG: --topology-manager-policy="none"
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181403    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181406    2577 flags.go:64] FLAG: --topology-manager-scope="container"
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181409    2577 flags.go:64] FLAG: --v="2"
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181413    2577 flags.go:64] FLAG: --version="false"
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181417    2577 flags.go:64] FLAG: --vmodule=""
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181421    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.181424    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181548    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181553    2577 feature_gate.go:328] unrecognized feature gate: Example2
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181555    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181558    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 23:02:07.187462 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181561    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181563    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181566    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181568    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181571    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181574    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181577    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181580    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181583    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181585    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181588    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181590    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181598    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181601    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181604    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181607    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181609    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181611    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181614    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 23:02:07.188063 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181616    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181619    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181621    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181624    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181626    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181628    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181631    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181633    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181636    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181639    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181642    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181646    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181649    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181652    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181654    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181656    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181659    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181661    2577 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181664    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181668    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 23:02:07.188629 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181671    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181673    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181676    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181678    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181681    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181684    2577 feature_gate.go:328] unrecognized feature gate: Example
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181691    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181694    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181696    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181698    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181701    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181703    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181706    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181708    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181710    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181713    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181715    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181718    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181720    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181722    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 23:02:07.189198 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181725    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181729    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181731    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181734    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181736    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181739    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181741    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181744    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181746    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181748    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181751    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181755    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181757    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181760    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181762    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181765    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181768    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181770    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181773    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181776    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 23:02:07.189680 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181779    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 23:02:07.190186 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181781    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 23:02:07.190186 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.181785    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 23:02:07.190186 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.182765    2577 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 23:02:07.190915 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.190893    2577 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 28 23:02:07.190951 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.190918    2577 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 28 23:02:07.190998 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.190989    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 23:02:07.190998 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.190997    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191001    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191004    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191007    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191010    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191013    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191015    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191018    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191020    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191023    2577 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191025    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191028    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191031    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191033    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191036    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191038    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191041    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191043    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191046    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 23:02:07.191054 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191049    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191052    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191056    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191060    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191063    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191066    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191068    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191071    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191074    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191076    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191079    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191087    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191090    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191093    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191095    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191098    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191100    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191103    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191105    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191108    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 23:02:07.191510 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191110    2577 feature_gate.go:328] unrecognized feature gate: Example
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191113    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191116    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191119    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191121    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191123    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191126    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191128    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191131    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191133    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191136    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191138    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191141    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191144    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191147    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191150    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191152    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191155    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191157    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191160    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 23:02:07.192028 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191162    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191165    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191167    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191169    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191172    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191175    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191178    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191180    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191182    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191185    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191187    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191189    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191192    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191194    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191196    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191199    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191201    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191204    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191206    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191208    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 23:02:07.192499 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191211    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 23:02:07.192993 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191215    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 23:02:07.192993 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191218    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 23:02:07.192993 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191221    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 23:02:07.192993 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191224    2577 feature_gate.go:328] unrecognized feature gate: Example2
May 28 23:02:07.192993 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191228    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 23:02:07.192993 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.191233    2577 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 23:02:07.192993 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191352    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 23:02:07.192993 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191356    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 23:02:07.192993 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191360    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 23:02:07.192993 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191363    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 23:02:07.192993 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191365    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 23:02:07.192993 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191368    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 23:02:07.192993 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191371    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 23:02:07.192993 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191374    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 23:02:07.192993 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191376    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191379    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191381    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191384    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191386    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191389    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191391    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191393    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191396    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191398    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191401    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191403    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191405    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191409    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191412    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191415    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191417    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191420    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191422    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191425    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 23:02:07.193364 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191427    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191430    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191432    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191435    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191438    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191442    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191445    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191448    2577 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191451    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191453    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191456    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191458    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191460    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191463    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191465    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191468    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191471    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191473    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191475    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 23:02:07.193842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191478    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191480    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191482    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191485    2577 feature_gate.go:328] unrecognized feature gate: Example2
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191487    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191490    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191492    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191494    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191497    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191499    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191502    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191504    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191506    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191509    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191511    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191514    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191517    2577 feature_gate.go:328] unrecognized feature gate: Example
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191520    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191523    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191525    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 23:02:07.194327 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191528    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191531    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191533    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191536    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191538    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191540    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191543    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191545    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191548    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191551    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191553    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191555    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191558    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191560    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191563    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191565    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191567    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191570    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 23:02:07.194821 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:07.191572    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 23:02:07.195485 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.191576    2577 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 23:02:07.195485 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.192458    2577 server.go:962] "Client rotation is on, will bootstrap in background"
May 28 23:02:07.195925 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.195561    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 28 23:02:07.196621 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.196610    2577 server.go:1019] "Starting client certificate rotation"
May 28 23:02:07.196726 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.196709    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 28 23:02:07.196759 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.196750    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 28 23:02:07.224123 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.224105    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 28 23:02:07.236753 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.236714    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 28 23:02:07.248496 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.248478    2577 log.go:25] "Validated CRI v1 runtime API"
May 28 23:02:07.254911 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.254890    2577 log.go:25] "Validated CRI v1 image API"
May 28 23:02:07.256675 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.256658    2577 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 28 23:02:07.261585 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.261566    2577 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 28 23:02:07.261713 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.261693    2577 fs.go:135] Filesystem UUIDs: map[2e7cd8d1-33c7-4e2f-b9b3-02d84bb653d3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 af3ae47d-8fdc-47e4-8699-704e789b646c:/dev/nvme0n1p4]
May 28 23:02:07.261761 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.261714    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 28 23:02:07.266043 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.265931    2577 manager.go:217] Machine: {Timestamp:2026-05-28 23:02:07.265421374 +0000 UTC m=+0.458852618 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101752 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2368dfa8364e39fc5f33a84b95522d SystemUUID:ec2368df-a836-4e39-fc5f-33a84b95522d BootID:4ae629ab-8769-4e59-a16f-7316d2c1177c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0f:6a:32:73:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0f:6a:32:73:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:9f:0d:0b:a3:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 28 23:02:07.266788 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.266777    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 28 23:02:07.266927 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.266913    2577 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 28 23:02:07.269023 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.268999    2577 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 28 23:02:07.269164 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.269027    2577 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-104.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 28 23:02:07.269212 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.269174    2577 topology_manager.go:138] "Creating topology manager with none policy"
May 28 23:02:07.269212 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.269182    2577 container_manager_linux.go:306] "Creating device plugin manager"
May 28 23:02:07.269212 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.269190    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 28 23:02:07.270193 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.270182    2577 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 28 23:02:07.272600 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.272588    2577 state_mem.go:36] "Initialized new in-memory state store"
May 28 23:02:07.272910 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.272899    2577 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 28 23:02:07.275641 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.275630    2577 kubelet.go:491] "Attempting to sync node with API server"
May 28 23:02:07.275695 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.275646    2577 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 28 23:02:07.275695 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.275658    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 28 23:02:07.275695 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.275670    2577 kubelet.go:397] "Adding apiserver pod source"
May 28 23:02:07.275695 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.275679    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 28 23:02:07.277040 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.277027    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 28 23:02:07.277094 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.277049    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 28 23:02:07.286926 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.286905    2577 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 28 23:02:07.287023 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.286943    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bd94z"
May 28 23:02:07.289122 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.289109    2577 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 28 23:02:07.290039 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.290027    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 28 23:02:07.290077 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.290045    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 28 23:02:07.290077 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.290051    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 28 23:02:07.290077 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.290057    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 28 23:02:07.290077 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.290067    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 28 23:02:07.290077 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.290076    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 28 23:02:07.290209 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.290086    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 28 23:02:07.290209 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.290093    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 28 23:02:07.290209 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.290100    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 28 23:02:07.290209 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.290105    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 28 23:02:07.290209 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.290116    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 28 23:02:07.290504 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.290496    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 28 23:02:07.291583 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.291572    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 28 23:02:07.291615 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.291587    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 28 23:02:07.293933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.293702    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bd94z"
May 28 23:02:07.303146 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.303119    2577 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-104.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 28 23:02:07.304171 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.304140    2577 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 28 23:02:07.304171 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.304162    2577 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-104.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 28 23:02:07.304301 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.304246    2577 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 28 23:02:07.304301 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.304288    2577 server.go:1295] "Started kubelet"
May 28 23:02:07.304401 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.304347    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 28 23:02:07.304502 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.304437    2577 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 28 23:02:07.304566 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.304526    2577 server_v1.go:47] "podresources" method="list" useActivePods=true
May 28 23:02:07.305011 ip-10-0-143-104 systemd[1]: Started Kubernetes Kubelet.
May 28 23:02:07.305592 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.305486    2577 server.go:317] "Adding debug handlers to kubelet server"
May 28 23:02:07.308711 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.308695    2577 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 28 23:02:07.314284 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.314266    2577 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 28 23:02:07.315447 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.315432    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 28 23:02:07.316555 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.316536    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 28 23:02:07.316698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.316682    2577 volume_manager.go:295] "The desired_state_of_world populator starts"
May 28 23:02:07.316749 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.316701    2577 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 28 23:02:07.316788 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.316749    2577 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 28 23:02:07.316824 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.316803    2577 reconstruct.go:97] "Volume reconstruction finished"
May 28 23:02:07.316824 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.316812    2577 reconciler.go:26] "Reconciler: start to sync state"
May 28 23:02:07.317506 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.317247    2577 factory.go:55] Registering systemd factory
May 28 23:02:07.317506 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.317243    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-104.ec2.internal\" not found"
May 28 23:02:07.317506 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.317266    2577 factory.go:223] Registration of the systemd container factory successfully
May 28 23:02:07.317788 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.317759    2577 factory.go:153] Registering CRI-O factory
May 28 23:02:07.317788 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.317782    2577 factory.go:223] Registration of the crio container factory successfully
May 28 23:02:07.317911 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.317837    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 28 23:02:07.317911 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.317860    2577 factory.go:103] Registering Raw factory
May 28 23:02:07.317911 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.317893    2577 manager.go:1196] Started watching for new ooms in manager
May 28 23:02:07.319140 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.319065    2577 manager.go:319] Starting recovery of all containers
May 28 23:02:07.323484 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.323462    2577 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 23:02:07.329934 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.329918    2577 manager.go:324] Recovery completed
May 28 23:02:07.334226 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.334201    2577 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 23:02:07.336201 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.336182    2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-104.ec2.internal\" not found" node="ip-10-0-143-104.ec2.internal"
May 28 23:02:07.336349 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.336335    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasSufficientMemory"
May 28 23:02:07.336396 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.336365    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasNoDiskPressure"
May 28 23:02:07.336396 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.336380    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasSufficientPID"
May 28 23:02:07.336917 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.336902    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 28 23:02:07.336959 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.336917    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 28 23:02:07.336959 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.336934    2577 state_mem.go:36] "Initialized new in-memory state store"
May 28 23:02:07.340329 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.340316    2577 policy_none.go:49] "None policy: Start"
May 28 23:02:07.340375 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.340333    2577 memory_manager.go:186] "Starting memorymanager" policy="None"
May 28 23:02:07.340375 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.340343    2577 state_mem.go:35] "Initializing new in-memory state store"
May 28 23:02:07.386454 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.386438    2577 manager.go:341] "Starting Device Plugin manager"
May 28 23:02:07.402000 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.386544    2577 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 28 23:02:07.402000 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.386560    2577 server.go:85] "Starting device plugin registration server"
May 28 23:02:07.402000 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.386774    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
May 28 23:02:07.402000 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.386785    2577 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 28 23:02:07.402000 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.386906    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 28 23:02:07.402000 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.386985    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 28 23:02:07.402000 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.386995    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 28 23:02:07.402000 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.387543    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 28 23:02:07.402000 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.387585    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-104.ec2.internal\" not found"
May 28 23:02:07.484986 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.484912    2577 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 28 23:02:07.486310 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.486297    2577 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 28 23:02:07.486372 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.486319    2577 status_manager.go:230] "Starting to sync pod status with apiserver"
May 28 23:02:07.486372 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.486336    2577 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 28 23:02:07.486372 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.486343    2577 kubelet.go:2451] "Starting kubelet main sync loop"
May 28 23:02:07.486487 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.486388    2577 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 28 23:02:07.486951 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.486856    2577 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 23:02:07.488051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.488034    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasSufficientMemory"
May 28 23:02:07.488142 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.488066    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasNoDiskPressure"
May 28 23:02:07.488142 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.488079    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasSufficientPID"
May 28 23:02:07.488142 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.488107    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-104.ec2.internal"
May 28 23:02:07.491986 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.491971    2577 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 23:02:07.495220 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.495206    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-104.ec2.internal"
May 28 23:02:07.495270 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.495229    2577 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-104.ec2.internal\": node \"ip-10-0-143-104.ec2.internal\" not found"
May 28 23:02:07.522489 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.522459    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-104.ec2.internal\" not found"
May 28 23:02:07.586772 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.586742    2577 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-104.ec2.internal"]
May 28 23:02:07.586859 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.586827    2577 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 23:02:07.587754 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.587731    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasSufficientMemory"
May 28 23:02:07.587851 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.587762    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasNoDiskPressure"
May 28 23:02:07.587851 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.587772    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasSufficientPID"
May 28 23:02:07.588905 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.588890    2577 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 23:02:07.589521 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.589506    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal"
May 28 23:02:07.589611 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.589543    2577 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 23:02:07.589611 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.589601    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasSufficientMemory"
May 28 23:02:07.589716 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.589635    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasNoDiskPressure"
May 28 23:02:07.589716 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.589651    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasSufficientPID"
May 28 23:02:07.590194 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.590171    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasSufficientMemory"
May 28 23:02:07.590298 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.590206    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasNoDiskPressure"
May 28 23:02:07.590298 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.590229    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasSufficientPID"
May 28 23:02:07.590704 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.590687    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-104.ec2.internal"
May 28 23:02:07.590762 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.590714    2577 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 23:02:07.595168 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.595152    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasSufficientMemory"
May 28 23:02:07.595252 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.595177    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasNoDiskPressure"
May 28 23:02:07.595252 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.595188    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeHasSufficientPID"
May 28 23:02:07.618164 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.618139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13878c3c3eb1cd66e339c8ce757ecaa5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal\" (UID: \"13878c3c3eb1cd66e339c8ce757ecaa5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal"
May 28 23:02:07.618254 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.618167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed64232379ece85cb86969c0ef54ec63-config\") pod \"kube-apiserver-proxy-ip-10-0-143-104.ec2.internal\" (UID: \"ed64232379ece85cb86969c0ef54ec63\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-104.ec2.internal"
May 28 23:02:07.618254 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.618186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13878c3c3eb1cd66e339c8ce757ecaa5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal\" (UID: \"13878c3c3eb1cd66e339c8ce757ecaa5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal"
May 28 23:02:07.623228 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.623213    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-104.ec2.internal\" not found"
May 28 23:02:07.630540 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.630523    2577 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-104.ec2.internal\" not found" node="ip-10-0-143-104.ec2.internal"
May 28 23:02:07.634981 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.634966    2577 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-104.ec2.internal\" not found" node="ip-10-0-143-104.ec2.internal"
May 28 23:02:07.719117 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.719083    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13878c3c3eb1cd66e339c8ce757ecaa5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal\" (UID: \"13878c3c3eb1cd66e339c8ce757ecaa5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal"
May 28 23:02:07.719219 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.719121    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13878c3c3eb1cd66e339c8ce757ecaa5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal\" (UID: \"13878c3c3eb1cd66e339c8ce757ecaa5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal"
May 28 23:02:07.719219 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.719142    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed64232379ece85cb86969c0ef54ec63-config\") pod \"kube-apiserver-proxy-ip-10-0-143-104.ec2.internal\" (UID: \"ed64232379ece85cb86969c0ef54ec63\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-104.ec2.internal"
May 28 23:02:07.719219 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.719185    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13878c3c3eb1cd66e339c8ce757ecaa5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal\" (UID: \"13878c3c3eb1cd66e339c8ce757ecaa5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal"
May 28 23:02:07.719219 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.719195    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13878c3c3eb1cd66e339c8ce757ecaa5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal\" (UID: \"13878c3c3eb1cd66e339c8ce757ecaa5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal"
May 28 23:02:07.719367 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.719189    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed64232379ece85cb86969c0ef54ec63-config\") pod \"kube-apiserver-proxy-ip-10-0-143-104.ec2.internal\" (UID: \"ed64232379ece85cb86969c0ef54ec63\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-104.ec2.internal"
May 28 23:02:07.724179 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.724159    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-104.ec2.internal\" not found"
May 28 23:02:07.824338 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.824253    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-104.ec2.internal\" not found"
May 28 23:02:07.924740 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:07.924709    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-104.ec2.internal\" not found"
May 28 23:02:07.932907 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.932884    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal"
May 28 23:02:07.937635 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:07.937618    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-104.ec2.internal"
May 28 23:02:08.025388 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:08.025353    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-104.ec2.internal\" not found"
May 28 23:02:08.125989 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:08.125906    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-104.ec2.internal\" not found"
May 28 23:02:08.196124 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.196096    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 28 23:02:08.196892 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.196275    2577 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 28 23:02:08.196892 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.196275    2577 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 28 23:02:08.205917 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.205898    2577 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 23:02:08.216837 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.216807    2577 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal"
May 28 23:02:08.229405 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.229383    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 28 23:02:08.231301 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.231289    2577 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-104.ec2.internal"
May 28 23:02:08.240929 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.240915    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 28 23:02:08.276089 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.276068    2577 apiserver.go:52] "Watching apiserver"
May 28 23:02:08.280847 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.280822    2577 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 23:02:08.281465 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.281449    2577 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 28 23:02:08.284329 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.284301    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-rhwv2","openshift-multus/multus-dfz7l","openshift-multus/network-metrics-daemon-mwvzm","openshift-ovn-kubernetes/ovnkube-node-w59bt","kube-system/kube-apiserver-proxy-ip-10-0-143-104.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v","openshift-cluster-node-tuning-operator/tuned-drr5k","openshift-image-registry/node-ca-cm8m8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal","openshift-multus/multus-additional-cni-plugins-zvj99","openshift-network-diagnostics/network-check-target-2kktk","openshift-network-operator/iptables-alerter-w4wxq","kube-system/konnectivity-agent-hs8lt"]
May 28 23:02:08.287486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.287461    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.289195 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.289174    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.289308 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.289291    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:08.289411 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:08.289389    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:08.289854 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.289836    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 28 23:02:08.289979 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.289889    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xtr4d\""
May 28 23:02:08.289979 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.289890    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 28 23:02:08.291490 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.291469    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.291598 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.291548    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 28 23:02:08.291598 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.291591    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 28 23:02:08.291977 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.291961    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 28 23:02:08.292109 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.291994    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 28 23:02:08.292175 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.292143    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wvj2f\""
May 28 23:02:08.293537 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.293494    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 28 23:02:08.293812 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.293797    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bfthc\""
May 28 23:02:08.293812 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.293805    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 28 23:02:08.293971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.293956    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.294211 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.294188    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 28 23:02:08.294300 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.294273    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 28 23:02:08.294347 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.294303    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 28 23:02:08.294389 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.294374    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 28 23:02:08.296206 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.296187    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rhwv2"
May 28 23:02:08.296291 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.296231    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 28 23:02:08.296291 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.296256    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 28 23:02:08.296554 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.296535    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 28 23:02:08.296654 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.296639    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8nq9w\""
May 28 23:02:08.298427 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.298406    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fj5pj\""
May 28 23:02:08.298532 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.298440    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 28 23:02:08.298770 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.298754    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cm8m8"
May 28 23:02:08.299277 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.299260    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 28 23:02:08.300836 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.300815    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 28 23:02:08.300964 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.300821    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 28 23:02:08.300964 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.300851    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zmkgv\""
May 28 23:02:08.300964 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.300886    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 28 23:02:08.301158 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.301144    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.303045 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.303022    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-27 22:57:07 +0000 UTC" deadline="2028-02-12 02:03:27.181547954 +0000 UTC"
May 28 23:02:08.303045 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.303043    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14979h1m18.878507279s"
May 28 23:02:08.303226 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.303215    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 28 23:02:08.303554 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.303540    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 28 23:02:08.303608 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.303588    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9nr5f\""
May 28 23:02:08.303703 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.303688    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:08.303787 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:08.303768    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:08.306296 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.306274    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w4wxq"
May 28 23:02:08.308253 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.308235    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 28 23:02:08.308507 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.308467    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 28 23:02:08.308606 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.308553    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9bhbq\""
May 28 23:02:08.308671 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.308645    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hs8lt"
May 28 23:02:08.308855 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.308834    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 28 23:02:08.310801 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.310773    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6dtnj\""
May 28 23:02:08.310912 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.310802    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 28 23:02:08.310912 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.310830    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 28 23:02:08.315590 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.315566    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 28 23:02:08.317845 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.317825    2577 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 28 23:02:08.322468 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-run-netns\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.322572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kchpv\" (UniqueName: \"kubernetes.io/projected/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-kube-api-access-kchpv\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.322572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/42519605-09ec-4431-9c90-a67e86597a9e-iptables-alerter-script\") pod \"iptables-alerter-w4wxq\" (UID: \"42519605-09ec-4431-9c90-a67e86597a9e\") " pod="openshift-network-operator/iptables-alerter-w4wxq"
May 28 23:02:08.322572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.322715 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-cnibin\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.322715 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-kubelet\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.322715 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42519605-09ec-4431-9c90-a67e86597a9e-host-slash\") pod \"iptables-alerter-w4wxq\" (UID: \"42519605-09ec-4431-9c90-a67e86597a9e\") " pod="openshift-network-operator/iptables-alerter-w4wxq"
May 28 23:02:08.322715 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-run\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.322715 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-multus-daemon-config\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.322715 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322703    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/364d7926-6000-4731-9989-15e6d74a44c6-ovn-node-metrics-cert\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.322994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-multus-socket-dir-parent\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.322994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-hostroot\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.322994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqgc4\" (UniqueName: \"kubernetes.io/projected/42519605-09ec-4431-9c90-a67e86597a9e-kube-api-access-bqgc4\") pod \"iptables-alerter-w4wxq\" (UID: \"42519605-09ec-4431-9c90-a67e86597a9e\") " pod="openshift-network-operator/iptables-alerter-w4wxq"
May 28 23:02:08.322994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ad0103a-d3c7-4e1a-953d-1555182e1170-konnectivity-ca\") pod \"konnectivity-agent-hs8lt\" (UID: \"9ad0103a-d3c7-4e1a-953d-1555182e1170\") " pod="kube-system/konnectivity-agent-hs8lt"
May 28 23:02:08.322994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-cni-netd\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.322994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/364d7926-6000-4731-9989-15e6d74a44c6-ovnkube-script-lib\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.322994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gv8k\" (UniqueName: \"kubernetes.io/projected/8856d7aa-86ca-4ef4-9d92-ceac3dd18292-kube-api-access-7gv8k\") pod \"node-resolver-rhwv2\" (UID: \"8856d7aa-86ca-4ef4-9d92-ceac3dd18292\") " pod="openshift-dns/node-resolver-rhwv2"
May 28 23:02:08.322994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.322994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-systemd\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.322994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-os-release\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.322994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-etc-kubernetes\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.322994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.322982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/321b5d1e-083e-45f0-99c7-066f9ec958d2-system-cni-dir\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-sys-fs\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323038    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz7df\" (UniqueName: \"kubernetes.io/projected/364d7926-6000-4731-9989-15e6d74a44c6-kube-api-access-qz7df\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-kubernetes\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-sysctl-d\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-host\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-multus-conf-dir\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323182    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-slash\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/321b5d1e-083e-45f0-99c7-066f9ec958d2-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-var-lib-kubelet\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-multus-cni-dir\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-run-k8s-cni-cncf-io\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g5vf\" (UniqueName: \"kubernetes.io/projected/321b5d1e-083e-45f0-99c7-066f9ec958d2-kube-api-access-7g5vf\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv7gj\" (UniqueName: \"kubernetes.io/projected/9559d46d-6176-4e25-a648-d11e8f9e99a6-kube-api-access-wv7gj\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/364d7926-6000-4731-9989-15e6d74a44c6-ovnkube-config\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.323397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-var-lib-cni-bin\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-run-multus-certs\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-systemd-units\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-log-socket\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8353cc62-2b97-4f45-8a3d-b871840ee8e8-host\") pod \"node-ca-cm8m8\" (UID: \"8353cc62-2b97-4f45-8a3d-b871840ee8e8\") " pod="openshift-image-registry/node-ca-cm8m8"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-cni-binary-copy\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs\") pod \"network-metrics-daemon-mwvzm\" (UID: \"0a871ef7-90af-4a70-96ae-7df74d7ca072\") " pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-var-lib-openvswitch\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-etc-openvswitch\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/321b5d1e-083e-45f0-99c7-066f9ec958d2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-tuned\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-node-log\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8353cc62-2b97-4f45-8a3d-b871840ee8e8-serviceca\") pod \"node-ca-cm8m8\" (UID: \"8353cc62-2b97-4f45-8a3d-b871840ee8e8\") " pod="openshift-image-registry/node-ca-cm8m8"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/321b5d1e-083e-45f0-99c7-066f9ec958d2-cnibin\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/321b5d1e-083e-45f0-99c7-066f9ec958d2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-var-lib-cni-multus\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.323924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9lws\" (UniqueName: \"kubernetes.io/projected/8353cc62-2b97-4f45-8a3d-b871840ee8e8-kube-api-access-r9lws\") pod \"node-ca-cm8m8\" (UID: \"8353cc62-2b97-4f45-8a3d-b871840ee8e8\") " pod="openshift-image-registry/node-ca-cm8m8"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-sysconfig\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-sys\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323824    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-lib-modules\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323849    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-var-lib-kubelet\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-run-netns\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-run-openvswitch\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8856d7aa-86ca-4ef4-9d92-ceac3dd18292-hosts-file\") pod \"node-resolver-rhwv2\" (UID: \"8856d7aa-86ca-4ef4-9d92-ceac3dd18292\") " pod="openshift-dns/node-resolver-rhwv2"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/321b5d1e-083e-45f0-99c7-066f9ec958d2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.323998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-device-dir\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-tmp\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324047    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwwvx\" (UniqueName: \"kubernetes.io/projected/0a871ef7-90af-4a70-96ae-7df74d7ca072-kube-api-access-xwwvx\") pod \"network-metrics-daemon-mwvzm\" (UID: \"0a871ef7-90af-4a70-96ae-7df74d7ca072\") " pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-run-ovn\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8856d7aa-86ca-4ef4-9d92-ceac3dd18292-tmp-dir\") pod \"node-resolver-rhwv2\" (UID: \"8856d7aa-86ca-4ef4-9d92-ceac3dd18292\") " pod="openshift-dns/node-resolver-rhwv2"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/321b5d1e-083e-45f0-99c7-066f9ec958d2-os-release\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-socket-dir\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.324483 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-registration-dir\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.324980 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74ggn\" (UniqueName: \"kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn\") pod \"network-check-target-2kktk\" (UID: \"9c10eeed-3826-4345-9e9b-6b8d25111763\") " pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:08.324980 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-system-cni-dir\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.324980 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-run-systemd\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.324980 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324282    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-cni-bin\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.324980 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/364d7926-6000-4731-9989-15e6d74a44c6-env-overrides\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.324980 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ad0103a-d3c7-4e1a-953d-1555182e1170-agent-certs\") pod \"konnectivity-agent-hs8lt\" (UID: \"9ad0103a-d3c7-4e1a-953d-1555182e1170\") " pod="kube-system/konnectivity-agent-hs8lt"
May 28 23:02:08.324980 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.324980 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-modprobe-d\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.324980 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-sysctl-conf\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.324980 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.324478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs5sx\" (UniqueName: \"kubernetes.io/projected/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-kube-api-access-rs5sx\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.327173 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.327153    2577 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 28 23:02:08.349884 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.349841    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zcwws"
May 28 23:02:08.358282 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.358250    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zcwws"
May 28 23:02:08.424946 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.424920    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-lib-modules\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.425031 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.424959    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-var-lib-kubelet\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.425031 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.424984    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-run-netns\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.425031 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425005    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-run-openvswitch\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.425157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425030    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8856d7aa-86ca-4ef4-9d92-ceac3dd18292-hosts-file\") pod \"node-resolver-rhwv2\" (UID: \"8856d7aa-86ca-4ef4-9d92-ceac3dd18292\") " pod="openshift-dns/node-resolver-rhwv2"
May 28 23:02:08.425157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425041    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-var-lib-kubelet\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.425157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425043    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-lib-modules\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.425157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425084    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/321b5d1e-083e-45f0-99c7-066f9ec958d2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.425157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425092    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-run-openvswitch\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.425157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425103    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8856d7aa-86ca-4ef4-9d92-ceac3dd18292-hosts-file\") pod \"node-resolver-rhwv2\" (UID: \"8856d7aa-86ca-4ef4-9d92-ceac3dd18292\") " pod="openshift-dns/node-resolver-rhwv2"
May 28 23:02:08.425157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425113    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-device-dir\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.425157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425130    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-tmp\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.425157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425091    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-run-netns\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.425157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425154    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xwwvx\" (UniqueName: \"kubernetes.io/projected/0a871ef7-90af-4a70-96ae-7df74d7ca072-kube-api-access-xwwvx\") pod \"network-metrics-daemon-mwvzm\" (UID: \"0a871ef7-90af-4a70-96ae-7df74d7ca072\") " pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425179    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-run-ovn\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425146    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-device-dir\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425207    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8856d7aa-86ca-4ef4-9d92-ceac3dd18292-tmp-dir\") pod \"node-resolver-rhwv2\" (UID: \"8856d7aa-86ca-4ef4-9d92-ceac3dd18292\") " pod="openshift-dns/node-resolver-rhwv2"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425230    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/321b5d1e-083e-45f0-99c7-066f9ec958d2-os-release\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425225    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-run-ovn\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425255    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-socket-dir\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425284    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-registration-dir\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425303    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/321b5d1e-083e-45f0-99c7-066f9ec958d2-os-release\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425308    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-74ggn\" (UniqueName: \"kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn\") pod \"network-check-target-2kktk\" (UID: \"9c10eeed-3826-4345-9e9b-6b8d25111763\") " pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425352    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-system-cni-dir\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425394    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-run-systemd\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425423    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-cni-bin\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425402    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425474    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-run-systemd\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425476    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-registration-dir\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425482    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-system-cni-dir\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425499    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/364d7926-6000-4731-9989-15e6d74a44c6-env-overrides\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.425572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425521    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8856d7aa-86ca-4ef4-9d92-ceac3dd18292-tmp-dir\") pod \"node-resolver-rhwv2\" (UID: \"8856d7aa-86ca-4ef4-9d92-ceac3dd18292\") " pod="openshift-dns/node-resolver-rhwv2"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425527    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-cni-bin\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425580    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ad0103a-d3c7-4e1a-953d-1555182e1170-agent-certs\") pod \"konnectivity-agent-hs8lt\" (UID: \"9ad0103a-d3c7-4e1a-953d-1555182e1170\") " pod="kube-system/konnectivity-agent-hs8lt"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425615    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425661    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425668    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-socket-dir\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425671    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/321b5d1e-083e-45f0-99c7-066f9ec958d2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425677    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-modprobe-d\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425708    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-sysctl-conf\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425774    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rs5sx\" (UniqueName: \"kubernetes.io/projected/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-kube-api-access-rs5sx\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425805    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-run-netns\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425814    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-sysctl-conf\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425814    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-modprobe-d\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425831    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kchpv\" (UniqueName: \"kubernetes.io/projected/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-kube-api-access-kchpv\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425881    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-run-netns\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425937    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/42519605-09ec-4431-9c90-a67e86597a9e-iptables-alerter-script\") pod \"iptables-alerter-w4wxq\" (UID: \"42519605-09ec-4431-9c90-a67e86597a9e\") " pod="openshift-network-operator/iptables-alerter-w4wxq"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425966    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.426527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.425995    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/364d7926-6000-4731-9989-15e6d74a44c6-env-overrides\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426001    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-cnibin\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426035    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-kubelet\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426062    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426075    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-kubelet\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426059    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42519605-09ec-4431-9c90-a67e86597a9e-host-slash\") pod \"iptables-alerter-w4wxq\" (UID: \"42519605-09ec-4431-9c90-a67e86597a9e\") " pod="openshift-network-operator/iptables-alerter-w4wxq"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426107    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42519605-09ec-4431-9c90-a67e86597a9e-host-slash\") pod \"iptables-alerter-w4wxq\" (UID: \"42519605-09ec-4431-9c90-a67e86597a9e\") " pod="openshift-network-operator/iptables-alerter-w4wxq"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426097    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-cnibin\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426122    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-run\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426156    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-multus-daemon-config\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426184    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/364d7926-6000-4731-9989-15e6d74a44c6-ovn-node-metrics-cert\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426210    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-multus-socket-dir-parent\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426159    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-run\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426278    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-multus-socket-dir-parent\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426324    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-hostroot\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426301    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-hostroot\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426429    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgc4\" (UniqueName: \"kubernetes.io/projected/42519605-09ec-4431-9c90-a67e86597a9e-kube-api-access-bqgc4\") pod \"iptables-alerter-w4wxq\" (UID: \"42519605-09ec-4431-9c90-a67e86597a9e\") " pod="openshift-network-operator/iptables-alerter-w4wxq"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426453    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ad0103a-d3c7-4e1a-953d-1555182e1170-konnectivity-ca\") pod \"konnectivity-agent-hs8lt\" (UID: \"9ad0103a-d3c7-4e1a-953d-1555182e1170\") " pod="kube-system/konnectivity-agent-hs8lt"
May 28 23:02:08.427333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426478    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-cni-netd\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426502    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/364d7926-6000-4731-9989-15e6d74a44c6-ovnkube-script-lib\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426537    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7gv8k\" (UniqueName: \"kubernetes.io/projected/8856d7aa-86ca-4ef4-9d92-ceac3dd18292-kube-api-access-7gv8k\") pod \"node-resolver-rhwv2\" (UID: \"8856d7aa-86ca-4ef4-9d92-ceac3dd18292\") " pod="openshift-dns/node-resolver-rhwv2"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426562    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426572    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/42519605-09ec-4431-9c90-a67e86597a9e-iptables-alerter-script\") pod \"iptables-alerter-w4wxq\" (UID: \"42519605-09ec-4431-9c90-a67e86597a9e\") " pod="openshift-network-operator/iptables-alerter-w4wxq"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426587    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-systemd\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426616    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-os-release\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426633    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-cni-netd\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426636    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-multus-daemon-config\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426642    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-etc-kubernetes\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426674    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/321b5d1e-083e-45f0-99c7-066f9ec958d2-system-cni-dir\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426687    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426700    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-sys-fs\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426725    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qz7df\" (UniqueName: \"kubernetes.io/projected/364d7926-6000-4731-9989-15e6d74a44c6-kube-api-access-qz7df\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426753    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-kubernetes\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426778    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-sysctl-d\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426803    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-host\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.428192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426837    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-multus-conf-dir\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426895    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/321b5d1e-083e-45f0-99c7-066f9ec958d2-system-cni-dir\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426898    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-slash\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426946    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-host-slash\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426945    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/321b5d1e-083e-45f0-99c7-066f9ec958d2-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.426987    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-var-lib-kubelet\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427014    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-multus-cni-dir\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427031    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ad0103a-d3c7-4e1a-953d-1555182e1170-konnectivity-ca\") pod \"konnectivity-agent-hs8lt\" (UID: \"9ad0103a-d3c7-4e1a-953d-1555182e1170\") " pod="kube-system/konnectivity-agent-hs8lt"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427044    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-run-k8s-cni-cncf-io\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427071    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7g5vf\" (UniqueName: \"kubernetes.io/projected/321b5d1e-083e-45f0-99c7-066f9ec958d2-kube-api-access-7g5vf\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427098    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-sys-fs\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427100    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/364d7926-6000-4731-9989-15e6d74a44c6-ovnkube-script-lib\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427121    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427149    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wv7gj\" (UniqueName: \"kubernetes.io/projected/9559d46d-6176-4e25-a648-d11e8f9e99a6-kube-api-access-wv7gj\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427160    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-kubernetes\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427176    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/364d7926-6000-4731-9989-15e6d74a44c6-ovnkube-config\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427201    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-var-lib-cni-bin\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.429051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427232    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-run-multus-certs\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427247    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-host\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427254    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-systemd-units\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427280    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-log-socket\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427296    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-os-release\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427304    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8353cc62-2b97-4f45-8a3d-b871840ee8e8-host\") pod \"node-ca-cm8m8\" (UID: \"8353cc62-2b97-4f45-8a3d-b871840ee8e8\") " pod="openshift-image-registry/node-ca-cm8m8"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427301    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-var-lib-kubelet\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427328    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/321b5d1e-083e-45f0-99c7-066f9ec958d2-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427348    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-multus-cni-dir\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427353    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-systemd\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427205    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-run-k8s-cni-cncf-io\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427386    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-cni-binary-copy\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427405    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9559d46d-6176-4e25-a648-d11e8f9e99a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427414    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs\") pod \"network-metrics-daemon-mwvzm\" (UID: \"0a871ef7-90af-4a70-96ae-7df74d7ca072\") " pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427427    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-sysctl-d\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427440    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-var-lib-openvswitch\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427465    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-etc-openvswitch\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427468    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-etc-kubernetes\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.429594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427490    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/321b5d1e-083e-45f0-99c7-066f9ec958d2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427495    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-run-multus-certs\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427518    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-tuned\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427543    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-node-log\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427556    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-var-lib-cni-bin\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427563    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8353cc62-2b97-4f45-8a3d-b871840ee8e8-serviceca\") pod \"node-ca-cm8m8\" (UID: \"8353cc62-2b97-4f45-8a3d-b871840ee8e8\") " pod="openshift-image-registry/node-ca-cm8m8"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427590    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/321b5d1e-083e-45f0-99c7-066f9ec958d2-cnibin\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427596    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-multus-conf-dir\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427610    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-log-socket\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427611    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/321b5d1e-083e-45f0-99c7-066f9ec958d2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427638    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8353cc62-2b97-4f45-8a3d-b871840ee8e8-host\") pod \"node-ca-cm8m8\" (UID: \"8353cc62-2b97-4f45-8a3d-b871840ee8e8\") " pod="openshift-image-registry/node-ca-cm8m8"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427676    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-var-lib-cni-multus\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427638    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-host-var-lib-cni-multus\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427719    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/321b5d1e-083e-45f0-99c7-066f9ec958d2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427715    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r9lws\" (UniqueName: \"kubernetes.io/projected/8353cc62-2b97-4f45-8a3d-b871840ee8e8-kube-api-access-r9lws\") pod \"node-ca-cm8m8\" (UID: \"8353cc62-2b97-4f45-8a3d-b871840ee8e8\") " pod="openshift-image-registry/node-ca-cm8m8"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427756    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-sysconfig\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427778    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-sys\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427845    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-sys\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.430143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427924    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/364d7926-6000-4731-9989-15e6d74a44c6-ovnkube-config\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.430796 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427986    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-sysconfig\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.430796 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427997    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-var-lib-openvswitch\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.430796 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.427547    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-systemd-units\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.430796 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.428044    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-node-log\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.430796 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:08.428064    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 23:02:08.430796 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:08.428135    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs podName:0a871ef7-90af-4a70-96ae-7df74d7ca072 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:08.928112566 +0000 UTC m=+2.121543810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs") pod "network-metrics-daemon-mwvzm" (UID: "0a871ef7-90af-4a70-96ae-7df74d7ca072") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 23:02:08.430796 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.428177    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-cni-binary-copy\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.430796 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.428224    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/364d7926-6000-4731-9989-15e6d74a44c6-etc-openvswitch\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.430796 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.428256    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/321b5d1e-083e-45f0-99c7-066f9ec958d2-cnibin\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.430796 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.428416    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8353cc62-2b97-4f45-8a3d-b871840ee8e8-serviceca\") pod \"node-ca-cm8m8\" (UID: \"8353cc62-2b97-4f45-8a3d-b871840ee8e8\") " pod="openshift-image-registry/node-ca-cm8m8"
May 28 23:02:08.430796 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.428742    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/321b5d1e-083e-45f0-99c7-066f9ec958d2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.430796 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.428785    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-tmp\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.430796 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.429168    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ad0103a-d3c7-4e1a-953d-1555182e1170-agent-certs\") pod \"konnectivity-agent-hs8lt\" (UID: \"9ad0103a-d3c7-4e1a-953d-1555182e1170\") " pod="kube-system/konnectivity-agent-hs8lt"
May 28 23:02:08.430796 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.430349    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/364d7926-6000-4731-9989-15e6d74a44c6-ovn-node-metrics-cert\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.430796 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.430767    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-etc-tuned\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.435509 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:08.435440    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 23:02:08.435509 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:08.435462    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 23:02:08.435509 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:08.435474    2577 projected.go:194] Error preparing data for projected volume kube-api-access-74ggn for pod openshift-network-diagnostics/network-check-target-2kktk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 23:02:08.435509 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:08.435532    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn podName:9c10eeed-3826-4345-9e9b-6b8d25111763 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:08.93551828 +0000 UTC m=+2.128949509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-74ggn" (UniqueName: "kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn") pod "network-check-target-2kktk" (UID: "9c10eeed-3826-4345-9e9b-6b8d25111763") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 23:02:08.436408 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:08.436378    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded64232379ece85cb86969c0ef54ec63.slice/crio-78ec87f142f7cfcbd663e8cc7ac5a7c8ff8b1ddccda354e878b9d2ea269465b6 WatchSource:0}: Error finding container 78ec87f142f7cfcbd663e8cc7ac5a7c8ff8b1ddccda354e878b9d2ea269465b6: Status 404 returned error can't find the container with id 78ec87f142f7cfcbd663e8cc7ac5a7c8ff8b1ddccda354e878b9d2ea269465b6
May 28 23:02:08.436614 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:08.436592    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13878c3c3eb1cd66e339c8ce757ecaa5.slice/crio-a372e03dd654e69ae1484d2ab3e3636fad0b2343d865b606d086f3de930850a8 WatchSource:0}: Error finding container a372e03dd654e69ae1484d2ab3e3636fad0b2343d865b606d086f3de930850a8: Status 404 returned error can't find the container with id a372e03dd654e69ae1484d2ab3e3636fad0b2343d865b606d086f3de930850a8
May 28 23:02:08.437191 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.437169    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwwvx\" (UniqueName: \"kubernetes.io/projected/0a871ef7-90af-4a70-96ae-7df74d7ca072-kube-api-access-xwwvx\") pod \"network-metrics-daemon-mwvzm\" (UID: \"0a871ef7-90af-4a70-96ae-7df74d7ca072\") " pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:08.440292 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.440266    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kchpv\" (UniqueName: \"kubernetes.io/projected/98206e77-e8fd-42da-bfb0-3fd1779ddd2e-kube-api-access-kchpv\") pod \"multus-dfz7l\" (UID: \"98206e77-e8fd-42da-bfb0-3fd1779ddd2e\") " pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.440456 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.440438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgc4\" (UniqueName: \"kubernetes.io/projected/42519605-09ec-4431-9c90-a67e86597a9e-kube-api-access-bqgc4\") pod \"iptables-alerter-w4wxq\" (UID: \"42519605-09ec-4431-9c90-a67e86597a9e\") " pod="openshift-network-operator/iptables-alerter-w4wxq"
May 28 23:02:08.440763 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.440739    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv7gj\" (UniqueName: \"kubernetes.io/projected/9559d46d-6176-4e25-a648-d11e8f9e99a6-kube-api-access-wv7gj\") pod \"aws-ebs-csi-driver-node-26l5v\" (UID: \"9559d46d-6176-4e25-a648-d11e8f9e99a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.440841 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.440785    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g5vf\" (UniqueName: \"kubernetes.io/projected/321b5d1e-083e-45f0-99c7-066f9ec958d2-kube-api-access-7g5vf\") pod \"multus-additional-cni-plugins-zvj99\" (UID: \"321b5d1e-083e-45f0-99c7-066f9ec958d2\") " pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.441355 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.440852    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gv8k\" (UniqueName: \"kubernetes.io/projected/8856d7aa-86ca-4ef4-9d92-ceac3dd18292-kube-api-access-7gv8k\") pod \"node-resolver-rhwv2\" (UID: \"8856d7aa-86ca-4ef4-9d92-ceac3dd18292\") " pod="openshift-dns/node-resolver-rhwv2"
May 28 23:02:08.441536 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.441363    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 23:02:08.442921 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.442851    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9lws\" (UniqueName: \"kubernetes.io/projected/8353cc62-2b97-4f45-8a3d-b871840ee8e8-kube-api-access-r9lws\") pod \"node-ca-cm8m8\" (UID: \"8353cc62-2b97-4f45-8a3d-b871840ee8e8\") " pod="openshift-image-registry/node-ca-cm8m8"
May 28 23:02:08.443791 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.443610    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs5sx\" (UniqueName: \"kubernetes.io/projected/0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb-kube-api-access-rs5sx\") pod \"tuned-drr5k\" (UID: \"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb\") " pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.444074 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.444057    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz7df\" (UniqueName: \"kubernetes.io/projected/364d7926-6000-4731-9989-15e6d74a44c6-kube-api-access-qz7df\") pod \"ovnkube-node-w59bt\" (UID: \"364d7926-6000-4731-9989-15e6d74a44c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.446202 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.446177    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hs8lt"
May 28 23:02:08.453478 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:08.453458    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad0103a_d3c7_4e1a_953d_1555182e1170.slice/crio-888805906e5f7c25977d6fb9f25b8db7340ec32dbaaa280b53b2103bd68d5451 WatchSource:0}: Error finding container 888805906e5f7c25977d6fb9f25b8db7340ec32dbaaa280b53b2103bd68d5451: Status 404 returned error can't find the container with id 888805906e5f7c25977d6fb9f25b8db7340ec32dbaaa280b53b2103bd68d5451
May 28 23:02:08.489418 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.489364    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hs8lt" event={"ID":"9ad0103a-d3c7-4e1a-953d-1555182e1170","Type":"ContainerStarted","Data":"888805906e5f7c25977d6fb9f25b8db7340ec32dbaaa280b53b2103bd68d5451"}
May 28 23:02:08.491367 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.491342    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-104.ec2.internal" event={"ID":"ed64232379ece85cb86969c0ef54ec63","Type":"ContainerStarted","Data":"78ec87f142f7cfcbd663e8cc7ac5a7c8ff8b1ddccda354e878b9d2ea269465b6"}
May 28 23:02:08.492212 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.492192    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal" event={"ID":"13878c3c3eb1cd66e339c8ce757ecaa5","Type":"ContainerStarted","Data":"a372e03dd654e69ae1484d2ab3e3636fad0b2343d865b606d086f3de930850a8"}
May 28 23:02:08.618537 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.618504    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-drr5k"
May 28 23:02:08.624363 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:08.624335    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1000e8_ffa7_412e_a0ba_d0717ba2bbbb.slice/crio-173d79cc20054534e7d7c47dc31d0c48a76d1563714e6ac3c955a5445209227e WatchSource:0}: Error finding container 173d79cc20054534e7d7c47dc31d0c48a76d1563714e6ac3c955a5445209227e: Status 404 returned error can't find the container with id 173d79cc20054534e7d7c47dc31d0c48a76d1563714e6ac3c955a5445209227e
May 28 23:02:08.640169 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.640150    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dfz7l"
May 28 23:02:08.645665 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:08.645640    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98206e77_e8fd_42da_bfb0_3fd1779ddd2e.slice/crio-597f5c0d3099ae8db9ce8deacdd8ac40541e87a7d33dd8fa00916a8bf7994714 WatchSource:0}: Error finding container 597f5c0d3099ae8db9ce8deacdd8ac40541e87a7d33dd8fa00916a8bf7994714: Status 404 returned error can't find the container with id 597f5c0d3099ae8db9ce8deacdd8ac40541e87a7d33dd8fa00916a8bf7994714
May 28 23:02:08.648520 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.648496    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:08.654157 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:08.654137    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364d7926_6000_4731_9989_15e6d74a44c6.slice/crio-49dd931978b38bacc3721788c7ecb08219eb1b04f2aadf80119b89faa41eb0fc WatchSource:0}: Error finding container 49dd931978b38bacc3721788c7ecb08219eb1b04f2aadf80119b89faa41eb0fc: Status 404 returned error can't find the container with id 49dd931978b38bacc3721788c7ecb08219eb1b04f2aadf80119b89faa41eb0fc
May 28 23:02:08.657904 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.657887    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v"
May 28 23:02:08.663518 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:08.663295    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9559d46d_6176_4e25_a648_d11e8f9e99a6.slice/crio-dd5c8b58020219de0a5eec881c9fc6f0f5441189ffa800d874a0f1b36bf022cd WatchSource:0}: Error finding container dd5c8b58020219de0a5eec881c9fc6f0f5441189ffa800d874a0f1b36bf022cd: Status 404 returned error can't find the container with id dd5c8b58020219de0a5eec881c9fc6f0f5441189ffa800d874a0f1b36bf022cd
May 28 23:02:08.674396 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.674373    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rhwv2"
May 28 23:02:08.680592 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:08.680570    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8856d7aa_86ca_4ef4_9d92_ceac3dd18292.slice/crio-3afc64a722322766e29beeeb8ccb40332d69756a497893e7610627f99aa8e37a WatchSource:0}: Error finding container 3afc64a722322766e29beeeb8ccb40332d69756a497893e7610627f99aa8e37a: Status 404 returned error can't find the container with id 3afc64a722322766e29beeeb8ccb40332d69756a497893e7610627f99aa8e37a
May 28 23:02:08.694932 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.694913    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cm8m8"
May 28 23:02:08.700575 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:08.700556    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8353cc62_2b97_4f45_8a3d_b871840ee8e8.slice/crio-30ab2682418aee2dcb5a02970ec7c0f85dac5cb933e0909b843399bd3780d3a2 WatchSource:0}: Error finding container 30ab2682418aee2dcb5a02970ec7c0f85dac5cb933e0909b843399bd3780d3a2: Status 404 returned error can't find the container with id 30ab2682418aee2dcb5a02970ec7c0f85dac5cb933e0909b843399bd3780d3a2
May 28 23:02:08.712978 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.712960    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvj99"
May 28 23:02:08.720271 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:08.719410    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321b5d1e_083e_45f0_99c7_066f9ec958d2.slice/crio-b8597783909e092757cd6de666640bd0d96ce4481aa581bd5e3d38076a2c9594 WatchSource:0}: Error finding container b8597783909e092757cd6de666640bd0d96ce4481aa581bd5e3d38076a2c9594: Status 404 returned error can't find the container with id b8597783909e092757cd6de666640bd0d96ce4481aa581bd5e3d38076a2c9594
May 28 23:02:08.740463 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.740440    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w4wxq"
May 28 23:02:08.745972 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:02:08.745943    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42519605_09ec_4431_9c90_a67e86597a9e.slice/crio-fb483f853f2f53673d175a12a49abac6c3d2d97860df1bf2554beca57883bb36 WatchSource:0}: Error finding container fb483f853f2f53673d175a12a49abac6c3d2d97860df1bf2554beca57883bb36: Status 404 returned error can't find the container with id fb483f853f2f53673d175a12a49abac6c3d2d97860df1bf2554beca57883bb36
May 28 23:02:08.931191 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:08.931107    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs\") pod \"network-metrics-daemon-mwvzm\" (UID: \"0a871ef7-90af-4a70-96ae-7df74d7ca072\") " pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:08.931356 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:08.931263    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 23:02:08.931356 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:08.931313    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs podName:0a871ef7-90af-4a70-96ae-7df74d7ca072 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:09.931300288 +0000 UTC m=+3.124731518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs") pod "network-metrics-daemon-mwvzm" (UID: "0a871ef7-90af-4a70-96ae-7df74d7ca072") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 23:02:09.032200 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.031920    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-74ggn\" (UniqueName: \"kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn\") pod \"network-check-target-2kktk\" (UID: \"9c10eeed-3826-4345-9e9b-6b8d25111763\") " pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:09.032200 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:09.032134    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 23:02:09.032200 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:09.032163    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 23:02:09.032200 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:09.032177    2577 projected.go:194] Error preparing data for projected volume kube-api-access-74ggn for pod openshift-network-diagnostics/network-check-target-2kktk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 23:02:09.032550 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:09.032238    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn podName:9c10eeed-3826-4345-9e9b-6b8d25111763 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:10.03221804 +0000 UTC m=+3.225649293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-74ggn" (UniqueName: "kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn") pod "network-check-target-2kktk" (UID: "9c10eeed-3826-4345-9e9b-6b8d25111763") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 23:02:09.144789 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.144758    2577 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 23:02:09.360122 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.359834    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 22:57:08 +0000 UTC" deadline="2028-03-15 15:52:58.618336092 +0000 UTC"
May 28 23:02:09.360122 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.359882    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15760h50m49.258458388s"
May 28 23:02:09.488077 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.488048    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:09.488284 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.488091    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:09.488284 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:09.488192    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:09.488641 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:09.488613    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:09.509551 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.509489    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w4wxq" event={"ID":"42519605-09ec-4431-9c90-a67e86597a9e","Type":"ContainerStarted","Data":"fb483f853f2f53673d175a12a49abac6c3d2d97860df1bf2554beca57883bb36"}
May 28 23:02:09.525898 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.525848    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvj99" event={"ID":"321b5d1e-083e-45f0-99c7-066f9ec958d2","Type":"ContainerStarted","Data":"b8597783909e092757cd6de666640bd0d96ce4481aa581bd5e3d38076a2c9594"}
May 28 23:02:09.544106 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.544069    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cm8m8" event={"ID":"8353cc62-2b97-4f45-8a3d-b871840ee8e8","Type":"ContainerStarted","Data":"30ab2682418aee2dcb5a02970ec7c0f85dac5cb933e0909b843399bd3780d3a2"}
May 28 23:02:09.544323 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.544304    2577 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 23:02:09.551000 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.550939    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt" event={"ID":"364d7926-6000-4731-9989-15e6d74a44c6","Type":"ContainerStarted","Data":"49dd931978b38bacc3721788c7ecb08219eb1b04f2aadf80119b89faa41eb0fc"}
May 28 23:02:09.565918 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.565886    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dfz7l" event={"ID":"98206e77-e8fd-42da-bfb0-3fd1779ddd2e","Type":"ContainerStarted","Data":"597f5c0d3099ae8db9ce8deacdd8ac40541e87a7d33dd8fa00916a8bf7994714"}
May 28 23:02:09.586560 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.585906    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-drr5k" event={"ID":"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb","Type":"ContainerStarted","Data":"173d79cc20054534e7d7c47dc31d0c48a76d1563714e6ac3c955a5445209227e"}
May 28 23:02:09.608257 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.608220    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rhwv2" event={"ID":"8856d7aa-86ca-4ef4-9d92-ceac3dd18292","Type":"ContainerStarted","Data":"3afc64a722322766e29beeeb8ccb40332d69756a497893e7610627f99aa8e37a"}
May 28 23:02:09.619792 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.619694    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v" event={"ID":"9559d46d-6176-4e25-a648-d11e8f9e99a6","Type":"ContainerStarted","Data":"dd5c8b58020219de0a5eec881c9fc6f0f5441189ffa800d874a0f1b36bf022cd"}
May 28 23:02:09.940244 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:09.940164    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs\") pod \"network-metrics-daemon-mwvzm\" (UID: \"0a871ef7-90af-4a70-96ae-7df74d7ca072\") " pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:09.940399 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:09.940354    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 23:02:09.940464 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:09.940421    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs podName:0a871ef7-90af-4a70-96ae-7df74d7ca072 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:11.940402085 +0000 UTC m=+5.133833315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs") pod "network-metrics-daemon-mwvzm" (UID: "0a871ef7-90af-4a70-96ae-7df74d7ca072") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 23:02:10.041149 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:10.040528    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-74ggn\" (UniqueName: \"kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn\") pod \"network-check-target-2kktk\" (UID: \"9c10eeed-3826-4345-9e9b-6b8d25111763\") " pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:10.041149 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:10.040688    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 23:02:10.041149 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:10.040707    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 23:02:10.041149 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:10.040719    2577 projected.go:194] Error preparing data for projected volume kube-api-access-74ggn for pod openshift-network-diagnostics/network-check-target-2kktk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 23:02:10.041149 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:10.040775    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn podName:9c10eeed-3826-4345-9e9b-6b8d25111763 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:12.040756928 +0000 UTC m=+5.234188166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-74ggn" (UniqueName: "kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn") pod "network-check-target-2kktk" (UID: "9c10eeed-3826-4345-9e9b-6b8d25111763") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 23:02:10.361102 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:10.360983    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 22:57:08 +0000 UTC" deadline="2027-10-23 18:53:28.592620782 +0000 UTC"
May 28 23:02:10.361102 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:10.361019    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12307h51m18.231605303s"
May 28 23:02:11.489473 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:11.489442    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:11.489927 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:11.489567    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:11.490016 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:11.489995    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:11.490124 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:11.490104    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:11.957491 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:11.957409    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs\") pod \"network-metrics-daemon-mwvzm\" (UID: \"0a871ef7-90af-4a70-96ae-7df74d7ca072\") " pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:11.957660 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:11.957572    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 23:02:11.957660 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:11.957649    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs podName:0a871ef7-90af-4a70-96ae-7df74d7ca072 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:15.957627395 +0000 UTC m=+9.151058639 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs") pod "network-metrics-daemon-mwvzm" (UID: "0a871ef7-90af-4a70-96ae-7df74d7ca072") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 23:02:12.057995 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:12.057961    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-74ggn\" (UniqueName: \"kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn\") pod \"network-check-target-2kktk\" (UID: \"9c10eeed-3826-4345-9e9b-6b8d25111763\") " pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:12.058167 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:12.058130    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 23:02:12.058167 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:12.058149    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 23:02:12.058167 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:12.058163    2577 projected.go:194] Error preparing data for projected volume kube-api-access-74ggn for pod openshift-network-diagnostics/network-check-target-2kktk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 23:02:12.058323 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:12.058217    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn podName:9c10eeed-3826-4345-9e9b-6b8d25111763 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:16.058199272 +0000 UTC m=+9.251630515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-74ggn" (UniqueName: "kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn") pod "network-check-target-2kktk" (UID: "9c10eeed-3826-4345-9e9b-6b8d25111763") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 23:02:13.487473 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:13.486991    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:13.487473 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:13.487109    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:13.488301 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:13.488151    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:13.488301 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:13.488265    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:15.488015 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:15.487980    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:15.488499 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:15.488093    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:15.488499 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:15.488156    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:15.488499 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:15.488282    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:15.992999 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:15.992964    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs\") pod \"network-metrics-daemon-mwvzm\" (UID: \"0a871ef7-90af-4a70-96ae-7df74d7ca072\") " pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:15.993181 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:15.993131    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 23:02:15.993248 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:15.993209    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs podName:0a871ef7-90af-4a70-96ae-7df74d7ca072 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:23.993190771 +0000 UTC m=+17.186622021 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs") pod "network-metrics-daemon-mwvzm" (UID: "0a871ef7-90af-4a70-96ae-7df74d7ca072") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 23:02:16.094150 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:16.094039    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-74ggn\" (UniqueName: \"kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn\") pod \"network-check-target-2kktk\" (UID: \"9c10eeed-3826-4345-9e9b-6b8d25111763\") " pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:16.094323 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:16.094229    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 23:02:16.094323 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:16.094257    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 23:02:16.094323 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:16.094272    2577 projected.go:194] Error preparing data for projected volume kube-api-access-74ggn for pod openshift-network-diagnostics/network-check-target-2kktk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 23:02:16.094481 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:16.094334    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn podName:9c10eeed-3826-4345-9e9b-6b8d25111763 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:24.094313523 +0000 UTC m=+17.287744779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-74ggn" (UniqueName: "kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn") pod "network-check-target-2kktk" (UID: "9c10eeed-3826-4345-9e9b-6b8d25111763") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 23:02:17.487700 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:17.487667    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:17.488125 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:17.487782    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:17.488324 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:17.488298    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:17.488417 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:17.488401    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:19.486651 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:19.486616    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:19.487058 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:19.486623    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:19.487058 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:19.486721    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:19.487058 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:19.486794    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:21.486961 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:21.486922    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:21.486961 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:21.486950    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:21.487406 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:21.487070    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:21.487406 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:21.487219    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:23.182368 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:23.182285    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-xtwjw"]
May 28 23:02:23.191061 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:23.191036    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:23.191214 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:23.191112    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtwjw" podUID="1f610866-a2a5-4395-bcad-928e420c46bb"
May 28 23:02:23.245567 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:23.245527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret\") pod \"global-pull-secret-syncer-xtwjw\" (UID: \"1f610866-a2a5-4395-bcad-928e420c46bb\") " pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:23.245738 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:23.245677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1f610866-a2a5-4395-bcad-928e420c46bb-kubelet-config\") pod \"global-pull-secret-syncer-xtwjw\" (UID: \"1f610866-a2a5-4395-bcad-928e420c46bb\") " pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:23.245738 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:23.245713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1f610866-a2a5-4395-bcad-928e420c46bb-dbus\") pod \"global-pull-secret-syncer-xtwjw\" (UID: \"1f610866-a2a5-4395-bcad-928e420c46bb\") " pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:23.346455 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:23.346422    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1f610866-a2a5-4395-bcad-928e420c46bb-kubelet-config\") pod \"global-pull-secret-syncer-xtwjw\" (UID: \"1f610866-a2a5-4395-bcad-928e420c46bb\") " pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:23.346455 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:23.346460    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1f610866-a2a5-4395-bcad-928e420c46bb-dbus\") pod \"global-pull-secret-syncer-xtwjw\" (UID: \"1f610866-a2a5-4395-bcad-928e420c46bb\") " pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:23.346707 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:23.346496    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret\") pod \"global-pull-secret-syncer-xtwjw\" (UID: \"1f610866-a2a5-4395-bcad-928e420c46bb\") " pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:23.346707 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:23.346583    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1f610866-a2a5-4395-bcad-928e420c46bb-kubelet-config\") pod \"global-pull-secret-syncer-xtwjw\" (UID: \"1f610866-a2a5-4395-bcad-928e420c46bb\") " pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:23.346707 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:23.346637    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 23:02:23.346707 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:23.346696    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret podName:1f610866-a2a5-4395-bcad-928e420c46bb nodeName:}" failed. No retries permitted until 2026-05-28 23:02:23.846676004 +0000 UTC m=+17.040107240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret") pod "global-pull-secret-syncer-xtwjw" (UID: "1f610866-a2a5-4395-bcad-928e420c46bb") : object "kube-system"/"original-pull-secret" not registered
May 28 23:02:23.346899 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:23.346748    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1f610866-a2a5-4395-bcad-928e420c46bb-dbus\") pod \"global-pull-secret-syncer-xtwjw\" (UID: \"1f610866-a2a5-4395-bcad-928e420c46bb\") " pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:23.486720 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:23.486643    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:23.486720 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:23.486663    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:23.486914 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:23.486750    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:23.486914 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:23.486892    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:23.850413 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:23.850320    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret\") pod \"global-pull-secret-syncer-xtwjw\" (UID: \"1f610866-a2a5-4395-bcad-928e420c46bb\") " pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:23.850584 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:23.850452    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 23:02:23.850584 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:23.850518    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret podName:1f610866-a2a5-4395-bcad-928e420c46bb nodeName:}" failed. No retries permitted until 2026-05-28 23:02:24.850502226 +0000 UTC m=+18.043933455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret") pod "global-pull-secret-syncer-xtwjw" (UID: "1f610866-a2a5-4395-bcad-928e420c46bb") : object "kube-system"/"original-pull-secret" not registered
May 28 23:02:24.051748 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:24.051704    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs\") pod \"network-metrics-daemon-mwvzm\" (UID: \"0a871ef7-90af-4a70-96ae-7df74d7ca072\") " pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:24.051959 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:24.051848    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 23:02:24.051959 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:24.051941    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs podName:0a871ef7-90af-4a70-96ae-7df74d7ca072 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:40.051924706 +0000 UTC m=+33.245355939 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs") pod "network-metrics-daemon-mwvzm" (UID: "0a871ef7-90af-4a70-96ae-7df74d7ca072") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 23:02:24.152516 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:24.152425    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-74ggn\" (UniqueName: \"kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn\") pod \"network-check-target-2kktk\" (UID: \"9c10eeed-3826-4345-9e9b-6b8d25111763\") " pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:24.152661 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:24.152622    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 23:02:24.152661 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:24.152647    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 23:02:24.152661 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:24.152660    2577 projected.go:194] Error preparing data for projected volume kube-api-access-74ggn for pod openshift-network-diagnostics/network-check-target-2kktk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 23:02:24.152790 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:24.152727    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn podName:9c10eeed-3826-4345-9e9b-6b8d25111763 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:40.152706047 +0000 UTC m=+33.346137281 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-74ggn" (UniqueName: "kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn") pod "network-check-target-2kktk" (UID: "9c10eeed-3826-4345-9e9b-6b8d25111763") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 23:02:24.857953 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:24.857898    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret\") pod \"global-pull-secret-syncer-xtwjw\" (UID: \"1f610866-a2a5-4395-bcad-928e420c46bb\") " pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:24.858405 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:24.858074    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 23:02:24.858405 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:24.858137    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret podName:1f610866-a2a5-4395-bcad-928e420c46bb nodeName:}" failed. No retries permitted until 2026-05-28 23:02:26.858119392 +0000 UTC m=+20.051550635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret") pod "global-pull-secret-syncer-xtwjw" (UID: "1f610866-a2a5-4395-bcad-928e420c46bb") : object "kube-system"/"original-pull-secret" not registered
May 28 23:02:25.487001 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:25.486969    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:25.487001 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:25.487000    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:25.487247 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:25.486969    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:25.487247 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:25.487080    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtwjw" podUID="1f610866-a2a5-4395-bcad-928e420c46bb"
May 28 23:02:25.487247 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:25.487173    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:25.487373 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:25.487264    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:26.871259 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:26.871215    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret\") pod \"global-pull-secret-syncer-xtwjw\" (UID: \"1f610866-a2a5-4395-bcad-928e420c46bb\") " pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:26.871748 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:26.871344    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 23:02:26.871748 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:26.871422    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret podName:1f610866-a2a5-4395-bcad-928e420c46bb nodeName:}" failed. No retries permitted until 2026-05-28 23:02:30.871401684 +0000 UTC m=+24.064832920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret") pod "global-pull-secret-syncer-xtwjw" (UID: "1f610866-a2a5-4395-bcad-928e420c46bb") : object "kube-system"/"original-pull-secret" not registered
May 28 23:02:27.487503 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:27.487467    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:27.487631 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:27.487565    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtwjw" podUID="1f610866-a2a5-4395-bcad-928e420c46bb"
May 28 23:02:27.487690 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:27.487643    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:27.487755 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:27.487733    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:27.487813 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:27.487776    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:27.487889 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:27.487823    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:28.657937 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.657752    2577 generic.go:358] "Generic (PLEG): container finished" podID="321b5d1e-083e-45f0-99c7-066f9ec958d2" containerID="f8ccb8c76b3e5e6e8d450976bd9ed74e5247202a11ad44b3b27b3fe7ec6bb674" exitCode=0
May 28 23:02:28.658698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.657839    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvj99" event={"ID":"321b5d1e-083e-45f0-99c7-066f9ec958d2","Type":"ContainerDied","Data":"f8ccb8c76b3e5e6e8d450976bd9ed74e5247202a11ad44b3b27b3fe7ec6bb674"}
May 28 23:02:28.659264 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.659239    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cm8m8" event={"ID":"8353cc62-2b97-4f45-8a3d-b871840ee8e8","Type":"ContainerStarted","Data":"cfbb30c6f6118d6e527f62855ff4a92a284dff33ab4576606bd9de9cf3167d01"}
May 28 23:02:28.661739 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.661692    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt" event={"ID":"364d7926-6000-4731-9989-15e6d74a44c6","Type":"ContainerStarted","Data":"a2b19f8f2769f753c4efb82537ef2553cbd3b11abb2441d2f626382b8687d520"}
May 28 23:02:28.661739 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.661718    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt" event={"ID":"364d7926-6000-4731-9989-15e6d74a44c6","Type":"ContainerStarted","Data":"b34d8bf16379cba9edef27df566d4b546110f664c325339423f9c9e956d708b5"}
May 28 23:02:28.661739 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.661727    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt" event={"ID":"364d7926-6000-4731-9989-15e6d74a44c6","Type":"ContainerStarted","Data":"1fb3f4099cfcadb36a6aaee7963d218152cd245971cf50c3defe4460c3411a2f"}
May 28 23:02:28.661739 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.661737    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt" event={"ID":"364d7926-6000-4731-9989-15e6d74a44c6","Type":"ContainerStarted","Data":"da7bbdc9032b1a958c970f37644f60e7b87ab860e2373f21da522230b369ce87"}
May 28 23:02:28.661940 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.661745    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt" event={"ID":"364d7926-6000-4731-9989-15e6d74a44c6","Type":"ContainerStarted","Data":"eaa5586dfc7057b5de692da98b28fe75c6cab579131386eee0f98fcc8064815e"}
May 28 23:02:28.661940 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.661754    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt" event={"ID":"364d7926-6000-4731-9989-15e6d74a44c6","Type":"ContainerStarted","Data":"05d764283754ed15a093643d73c2ee701461c44898f106ba9f56e3091e08800a"}
May 28 23:02:28.662886 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.662845    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dfz7l" event={"ID":"98206e77-e8fd-42da-bfb0-3fd1779ddd2e","Type":"ContainerStarted","Data":"63532eb2296a7683cce284823a0dc2d3458722337b39430cc6bebca3aa711def"}
May 28 23:02:28.664188 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.664163    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-drr5k" event={"ID":"0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb","Type":"ContainerStarted","Data":"71f8c0bc31d2bea365840b26358eec953daab29b113efcd6f2518cdbb11aa793"}
May 28 23:02:28.665461 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.665443    2577 generic.go:358] "Generic (PLEG): container finished" podID="13878c3c3eb1cd66e339c8ce757ecaa5" containerID="9ba621dd0aabf306415989eabb013ef3cffbce63175ec6707efbdde80668c1f3" exitCode=0
May 28 23:02:28.665540 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.665496    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal" event={"ID":"13878c3c3eb1cd66e339c8ce757ecaa5","Type":"ContainerDied","Data":"9ba621dd0aabf306415989eabb013ef3cffbce63175ec6707efbdde80668c1f3"}
May 28 23:02:28.666660 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.666629    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rhwv2" event={"ID":"8856d7aa-86ca-4ef4-9d92-ceac3dd18292","Type":"ContainerStarted","Data":"155832a81392e4c9e56452082dcbaa3c5c9a1e13c081a89e913d78774b148e6e"}
May 28 23:02:28.667788 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.667771    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v" event={"ID":"9559d46d-6176-4e25-a648-d11e8f9e99a6","Type":"ContainerStarted","Data":"19bfa0bdd2c7677e6c468b373fa3333f196af8f17510448068d17f9e98955d90"}
May 28 23:02:28.668878 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.668846    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hs8lt" event={"ID":"9ad0103a-d3c7-4e1a-953d-1555182e1170","Type":"ContainerStarted","Data":"8caea40987e6804e093635731622016d8f9f4630afc05f97c01ecaff29406235"}
May 28 23:02:28.669991 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.669971    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-104.ec2.internal" event={"ID":"ed64232379ece85cb86969c0ef54ec63","Type":"ContainerStarted","Data":"decf3007dc4a2336e776e125e5fee23adac10e06fada76f1615aab1941611468"}
May 28 23:02:28.699908 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.699843    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-rhwv2" podStartSLOduration=2.659247373 podStartE2EDuration="21.699830858s" podCreationTimestamp="2026-05-28 23:02:07 +0000 UTC" firstStartedPulling="2026-05-28 23:02:08.682300884 +0000 UTC m=+1.875732117" lastFinishedPulling="2026-05-28 23:02:27.722884372 +0000 UTC m=+20.916315602" observedRunningTime="2026-05-28 23:02:28.699300092 +0000 UTC m=+21.892731345" watchObservedRunningTime="2026-05-28 23:02:28.699830858 +0000 UTC m=+21.893262107"
May 28 23:02:28.714970 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.714920    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cm8m8" podStartSLOduration=2.74068809 podStartE2EDuration="21.714888526s" podCreationTimestamp="2026-05-28 23:02:07 +0000 UTC" firstStartedPulling="2026-05-28 23:02:08.701917862 +0000 UTC m=+1.895349092" lastFinishedPulling="2026-05-28 23:02:27.676118294 +0000 UTC m=+20.869549528" observedRunningTime="2026-05-28 23:02:28.714213891 +0000 UTC m=+21.907645143" watchObservedRunningTime="2026-05-28 23:02:28.714888526 +0000 UTC m=+21.908319778"
May 28 23:02:28.730938 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.730881    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hs8lt" podStartSLOduration=2.938575321 podStartE2EDuration="21.730851954s" podCreationTimestamp="2026-05-28 23:02:07 +0000 UTC" firstStartedPulling="2026-05-28 23:02:08.455024748 +0000 UTC m=+1.648455978" lastFinishedPulling="2026-05-28 23:02:27.247301377 +0000 UTC m=+20.440732611" observedRunningTime="2026-05-28 23:02:28.730341772 +0000 UTC m=+21.923773020" watchObservedRunningTime="2026-05-28 23:02:28.730851954 +0000 UTC m=+21.924283205"
May 28 23:02:28.763882 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.763815    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-drr5k" podStartSLOduration=2.667533004 podStartE2EDuration="21.763796955s" podCreationTimestamp="2026-05-28 23:02:07 +0000 UTC" firstStartedPulling="2026-05-28 23:02:08.625917066 +0000 UTC m=+1.819348295" lastFinishedPulling="2026-05-28 23:02:27.722181005 +0000 UTC m=+20.915612246" observedRunningTime="2026-05-28 23:02:28.747094888 +0000 UTC m=+21.940526136" watchObservedRunningTime="2026-05-28 23:02:28.763796955 +0000 UTC m=+21.957228208"
May 28 23:02:28.764531 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.764499    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-104.ec2.internal" podStartSLOduration=20.764489721 podStartE2EDuration="20.764489721s" podCreationTimestamp="2026-05-28 23:02:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:02:28.763960253 +0000 UTC m=+21.957391489" watchObservedRunningTime="2026-05-28 23:02:28.764489721 +0000 UTC m=+21.957920969"
May 28 23:02:28.781385 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:28.781343    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-dfz7l" podStartSLOduration=2.595683588 podStartE2EDuration="21.781331246s" podCreationTimestamp="2026-05-28 23:02:07 +0000 UTC" firstStartedPulling="2026-05-28 23:02:08.647068199 +0000 UTC m=+1.840499428" lastFinishedPulling="2026-05-28 23:02:27.832715846 +0000 UTC m=+21.026147086" observedRunningTime="2026-05-28 23:02:28.781220244 +0000 UTC m=+21.974651496" watchObservedRunningTime="2026-05-28 23:02:28.781331246 +0000 UTC m=+21.974762497"
May 28 23:02:29.405902 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:29.405856    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 28 23:02:29.486684 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:29.486586    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:29.486842 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:29.486712    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtwjw" podUID="1f610866-a2a5-4395-bcad-928e420c46bb"
May 28 23:02:29.486842 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:29.486586    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:29.486842 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:29.486815    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:29.486842 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:29.486586    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:29.487022 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:29.486903    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:29.674345 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:29.674307    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v" event={"ID":"9559d46d-6176-4e25-a648-d11e8f9e99a6","Type":"ContainerStarted","Data":"3d55a6ecbd9183cde5afab03a86fd29d8e36788a0fcf5866a964933215547c5c"}
May 28 23:02:29.675826 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:29.675796    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w4wxq" event={"ID":"42519605-09ec-4431-9c90-a67e86597a9e","Type":"ContainerStarted","Data":"046f476288074e157b2e2f7eadfaddd0b31051b89b06177acef2d56cd1cb3788"}
May 28 23:02:29.708285 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:29.708235    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w4wxq" podStartSLOduration=4.208174697 podStartE2EDuration="22.708220584s" podCreationTimestamp="2026-05-28 23:02:07 +0000 UTC" firstStartedPulling="2026-05-28 23:02:08.747252426 +0000 UTC m=+1.940683658" lastFinishedPulling="2026-05-28 23:02:27.247298301 +0000 UTC m=+20.440729545" observedRunningTime="2026-05-28 23:02:29.690633479 +0000 UTC m=+22.884064732" watchObservedRunningTime="2026-05-28 23:02:29.708220584 +0000 UTC m=+22.901651835"
May 28 23:02:30.348008 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:30.347969    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hs8lt"
May 28 23:02:30.348675 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:30.348652    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hs8lt"
May 28 23:02:30.401084 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:30.400962    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-28T23:02:29.405900364Z","UUID":"061853d0-2afa-4216-b711-49fe50d03a43","Handler":null,"Name":"","Endpoint":""}
May 28 23:02:30.404423 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:30.404215    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 28 23:02:30.404423 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:30.404247    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 28 23:02:30.680076 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:30.680043    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal" event={"ID":"13878c3c3eb1cd66e339c8ce757ecaa5","Type":"ContainerStarted","Data":"6e8ccc91968956173c55a581f25b4c757f2580d8852ba5a91ec62db0566ee3c9"}
May 28 23:02:30.682108 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:30.682079    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v" event={"ID":"9559d46d-6176-4e25-a648-d11e8f9e99a6","Type":"ContainerStarted","Data":"d0ed07003787c5477288332bfad9cf85ab70b169d8e38f21563a84c401fead79"}
May 28 23:02:30.682260 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:30.682237    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hs8lt"
May 28 23:02:30.682708 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:30.682686    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hs8lt"
May 28 23:02:30.695689 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:30.695649    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-104.ec2.internal" podStartSLOduration=22.695637332 podStartE2EDuration="22.695637332s" podCreationTimestamp="2026-05-28 23:02:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:02:30.69514099 +0000 UTC m=+23.888572241" watchObservedRunningTime="2026-05-28 23:02:30.695637332 +0000 UTC m=+23.889068583"
May 28 23:02:30.727027 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:30.726975    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-26l5v" podStartSLOduration=2.038167483 podStartE2EDuration="23.726958488s" podCreationTimestamp="2026-05-28 23:02:07 +0000 UTC" firstStartedPulling="2026-05-28 23:02:08.664744655 +0000 UTC m=+1.858175885" lastFinishedPulling="2026-05-28 23:02:30.353535661 +0000 UTC m=+23.546966890" observedRunningTime="2026-05-28 23:02:30.726203174 +0000 UTC m=+23.919634427" watchObservedRunningTime="2026-05-28 23:02:30.726958488 +0000 UTC m=+23.920389742"
May 28 23:02:30.903579 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:30.903491    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret\") pod \"global-pull-secret-syncer-xtwjw\" (UID: \"1f610866-a2a5-4395-bcad-928e420c46bb\") " pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:30.903732 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:30.903602    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 23:02:30.903732 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:30.903658    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret podName:1f610866-a2a5-4395-bcad-928e420c46bb nodeName:}" failed. No retries permitted until 2026-05-28 23:02:38.903640467 +0000 UTC m=+32.097071708 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret") pod "global-pull-secret-syncer-xtwjw" (UID: "1f610866-a2a5-4395-bcad-928e420c46bb") : object "kube-system"/"original-pull-secret" not registered
May 28 23:02:31.487345 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:31.487304    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:31.487345 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:31.487345    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:31.487575 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:31.487432    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtwjw" podUID="1f610866-a2a5-4395-bcad-928e420c46bb"
May 28 23:02:31.487575 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:31.487477    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:31.487575 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:31.487554    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:31.487701 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:31.487585    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:31.687252 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:31.687207    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt" event={"ID":"364d7926-6000-4731-9989-15e6d74a44c6","Type":"ContainerStarted","Data":"a28227e46c5473f0b2798737aca57127fd1f648391a128f2ec13f7fbec5c5623"}
May 28 23:02:33.487297 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:33.487048    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:33.488000 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:33.487054    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:33.488000 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:33.487311    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtwjw" podUID="1f610866-a2a5-4395-bcad-928e420c46bb"
May 28 23:02:33.488000 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:33.487120    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:33.488000 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:33.487390    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:33.488000 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:33.487464    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:33.693317 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:33.693279    2577 generic.go:358] "Generic (PLEG): container finished" podID="321b5d1e-083e-45f0-99c7-066f9ec958d2" containerID="6a87c2108737412c6668e86758410a7b3a21b06a94d34d322017b78886cedbb5" exitCode=0
May 28 23:02:33.693455 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:33.693354    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvj99" event={"ID":"321b5d1e-083e-45f0-99c7-066f9ec958d2","Type":"ContainerDied","Data":"6a87c2108737412c6668e86758410a7b3a21b06a94d34d322017b78886cedbb5"}
May 28 23:02:33.696777 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:33.696749    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt" event={"ID":"364d7926-6000-4731-9989-15e6d74a44c6","Type":"ContainerStarted","Data":"d3574cc930196f6ac8eaf640883965f3cf9ba9ebffd00592e615846555543a33"}
May 28 23:02:33.697090 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:33.697056    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:33.697090 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:33.697083    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:33.712793 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:33.712770    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:33.741182 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:33.741078    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt" podStartSLOduration=7.634839683 podStartE2EDuration="26.741064315s" podCreationTimestamp="2026-05-28 23:02:07 +0000 UTC" firstStartedPulling="2026-05-28 23:02:08.656570481 +0000 UTC m=+1.850001711" lastFinishedPulling="2026-05-28 23:02:27.762795112 +0000 UTC m=+20.956226343" observedRunningTime="2026-05-28 23:02:33.740299784 +0000 UTC m=+26.933731035" watchObservedRunningTime="2026-05-28 23:02:33.741064315 +0000 UTC m=+26.934495567"
May 28 23:02:34.699271 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:34.699243    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:34.714713 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:34.714690    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:02:34.860190 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:34.860154    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2kktk"]
May 28 23:02:34.860293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:34.860272    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:34.860424 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:34.860345    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:34.863474 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:34.863420    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xtwjw"]
May 28 23:02:34.863564 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:34.863546    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:34.863698 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:34.863676    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtwjw" podUID="1f610866-a2a5-4395-bcad-928e420c46bb"
May 28 23:02:34.864122 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:34.864097    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mwvzm"]
May 28 23:02:34.864232 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:34.864218    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:34.864345 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:34.864325    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:35.702011 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:35.701964    2577 generic.go:358] "Generic (PLEG): container finished" podID="321b5d1e-083e-45f0-99c7-066f9ec958d2" containerID="58db4796a57704f25708caed50379fa62502332248e4be8fb9ad51f1f106ee5c" exitCode=0
May 28 23:02:35.702426 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:35.702056    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvj99" event={"ID":"321b5d1e-083e-45f0-99c7-066f9ec958d2","Type":"ContainerDied","Data":"58db4796a57704f25708caed50379fa62502332248e4be8fb9ad51f1f106ee5c"}
May 28 23:02:36.487319 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:36.487146    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:36.487403 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:36.487200    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:36.487448 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:36.487402    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtwjw" podUID="1f610866-a2a5-4395-bcad-928e420c46bb"
May 28 23:02:36.487497 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:36.487480    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:36.487533 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:36.487216    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:36.487561 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:36.487547    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:36.706103 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:36.706014    2577 generic.go:358] "Generic (PLEG): container finished" podID="321b5d1e-083e-45f0-99c7-066f9ec958d2" containerID="628455a4bfed27841f7fcdce2005fcc539c60a1fbce0de5dc7c7a2be9316ca8e" exitCode=0
May 28 23:02:36.706556 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:36.706098    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvj99" event={"ID":"321b5d1e-083e-45f0-99c7-066f9ec958d2","Type":"ContainerDied","Data":"628455a4bfed27841f7fcdce2005fcc539c60a1fbce0de5dc7c7a2be9316ca8e"}
May 28 23:02:38.486580 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:38.486547    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:38.487368 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:38.486547    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:38.487368 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:38.486685    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:38.487368 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:38.486767    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtwjw" podUID="1f610866-a2a5-4395-bcad-928e420c46bb"
May 28 23:02:38.487368 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:38.486814    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:38.487368 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:38.486899    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:38.960191 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:38.960152    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret\") pod \"global-pull-secret-syncer-xtwjw\" (UID: \"1f610866-a2a5-4395-bcad-928e420c46bb\") " pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:38.960375 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:38.960333    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 23:02:38.960437 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:38.960399    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret podName:1f610866-a2a5-4395-bcad-928e420c46bb nodeName:}" failed. No retries permitted until 2026-05-28 23:02:54.960380404 +0000 UTC m=+48.153811636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret") pod "global-pull-secret-syncer-xtwjw" (UID: "1f610866-a2a5-4395-bcad-928e420c46bb") : object "kube-system"/"original-pull-secret" not registered
May 28 23:02:40.070754 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.070716    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs\") pod \"network-metrics-daemon-mwvzm\" (UID: \"0a871ef7-90af-4a70-96ae-7df74d7ca072\") " pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:40.071471 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:40.070906    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 23:02:40.071471 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:40.070981    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs podName:0a871ef7-90af-4a70-96ae-7df74d7ca072 nodeName:}" failed. No retries permitted until 2026-05-28 23:03:12.07096076 +0000 UTC m=+65.264391997 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs") pod "network-metrics-daemon-mwvzm" (UID: "0a871ef7-90af-4a70-96ae-7df74d7ca072") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 23:02:40.172136 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.172098    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-74ggn\" (UniqueName: \"kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn\") pod \"network-check-target-2kktk\" (UID: \"9c10eeed-3826-4345-9e9b-6b8d25111763\") " pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:40.172326 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:40.172268    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 23:02:40.172326 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:40.172286    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 23:02:40.172326 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:40.172298    2577 projected.go:194] Error preparing data for projected volume kube-api-access-74ggn for pod openshift-network-diagnostics/network-check-target-2kktk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 23:02:40.172468 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:40.172353    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn podName:9c10eeed-3826-4345-9e9b-6b8d25111763 nodeName:}" failed. No retries permitted until 2026-05-28 23:03:12.172334763 +0000 UTC m=+65.365766008 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-74ggn" (UniqueName: "kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn") pod "network-check-target-2kktk" (UID: "9c10eeed-3826-4345-9e9b-6b8d25111763") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 23:02:40.487453 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.487418    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:40.487602 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.487418    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:40.487602 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:40.487538    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2kktk" podUID="9c10eeed-3826-4345-9e9b-6b8d25111763"
May 28 23:02:40.487670 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.487435    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:40.487670 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:40.487605    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mwvzm" podUID="0a871ef7-90af-4a70-96ae-7df74d7ca072"
May 28 23:02:40.487728 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:40.487708    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtwjw" podUID="1f610866-a2a5-4395-bcad-928e420c46bb"
May 28 23:02:40.628274 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.628185    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-104.ec2.internal" event="NodeReady"
May 28 23:02:40.628414 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.628377    2577 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 28 23:02:40.672409 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.672376    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n77fp"]
May 28 23:02:40.692768 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.692736    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-47c8w"]
May 28 23:02:40.692960 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.692937    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n77fp"
May 28 23:02:40.696191 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.696167    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-94qps\""
May 28 23:02:40.696325 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.696208    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 28 23:02:40.696325 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.696167    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 28 23:02:40.704729 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.704706    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n77fp"]
May 28 23:02:40.704832 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.704739    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-47c8w"]
May 28 23:02:40.704898 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.704830    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:02:40.707429 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.707405    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 28 23:02:40.707577 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.707559    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 28 23:02:40.708103 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.708081    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v6j6q\""
May 28 23:02:40.708350 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.708335    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 28 23:02:40.776114 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.776083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-config-volume\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:40.776279 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.776127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:40.776279 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.776153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-tmp-dir\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:40.776279 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.776179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert\") pod \"ingress-canary-47c8w\" (UID: \"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77\") " pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:02:40.776279 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.776209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp44r\" (UniqueName: \"kubernetes.io/projected/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-kube-api-access-wp44r\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:40.776430 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.776300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dt59\" (UniqueName: \"kubernetes.io/projected/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-kube-api-access-2dt59\") pod \"ingress-canary-47c8w\" (UID: \"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77\") " pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:02:40.877544 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.877502    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:40.877544 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.877552    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-tmp-dir\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:40.877782 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.877584    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert\") pod \"ingress-canary-47c8w\" (UID: \"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77\") " pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:02:40.877782 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.877614    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wp44r\" (UniqueName: \"kubernetes.io/projected/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-kube-api-access-wp44r\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:40.877782 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.877647    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2dt59\" (UniqueName: \"kubernetes.io/projected/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-kube-api-access-2dt59\") pod \"ingress-canary-47c8w\" (UID: \"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77\") " pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:02:40.877782 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:40.877682    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 23:02:40.877782 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:40.877758    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls podName:0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b nodeName:}" failed. No retries permitted until 2026-05-28 23:02:41.377736205 +0000 UTC m=+34.571167462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls") pod "dns-default-n77fp" (UID: "0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b") : secret "dns-default-metrics-tls" not found
May 28 23:02:40.877782 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:40.877765    2577 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 23:02:40.878090 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:40.877835    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert podName:ba8f16cc-755b-4dc9-a1b5-3c33e7149c77 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:41.377817735 +0000 UTC m=+34.571248970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert") pod "ingress-canary-47c8w" (UID: "ba8f16cc-755b-4dc9-a1b5-3c33e7149c77") : secret "canary-serving-cert" not found
May 28 23:02:40.878090 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.877693    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-config-volume\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:40.878090 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.877990    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-tmp-dir\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:40.878382 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.878341    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-config-volume\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:40.888758 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.888735    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp44r\" (UniqueName: \"kubernetes.io/projected/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-kube-api-access-wp44r\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:40.888967 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:40.888940    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dt59\" (UniqueName: \"kubernetes.io/projected/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-kube-api-access-2dt59\") pod \"ingress-canary-47c8w\" (UID: \"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77\") " pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:02:41.381792 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:41.381749    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:41.382372 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:41.381807    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert\") pod \"ingress-canary-47c8w\" (UID: \"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77\") " pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:02:41.382372 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:41.381936    2577 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 23:02:41.382372 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:41.381938    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 23:02:41.382372 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:41.381997    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert podName:ba8f16cc-755b-4dc9-a1b5-3c33e7149c77 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:42.381978369 +0000 UTC m=+35.575409616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert") pod "ingress-canary-47c8w" (UID: "ba8f16cc-755b-4dc9-a1b5-3c33e7149c77") : secret "canary-serving-cert" not found
May 28 23:02:41.382372 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:41.382019    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls podName:0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b nodeName:}" failed. No retries permitted until 2026-05-28 23:02:42.382003857 +0000 UTC m=+35.575435110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls") pod "dns-default-n77fp" (UID: "0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b") : secret "dns-default-metrics-tls" not found
May 28 23:02:42.391345 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:42.391307    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert\") pod \"ingress-canary-47c8w\" (UID: \"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77\") " pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:02:42.391717 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:42.391378    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:42.391717 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:42.391460    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 23:02:42.391717 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:42.391459    2577 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 23:02:42.391717 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:42.391521    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls podName:0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b nodeName:}" failed. No retries permitted until 2026-05-28 23:02:44.391505002 +0000 UTC m=+37.584936238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls") pod "dns-default-n77fp" (UID: "0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b") : secret "dns-default-metrics-tls" not found
May 28 23:02:42.391717 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:42.391541    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert podName:ba8f16cc-755b-4dc9-a1b5-3c33e7149c77 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:44.391531924 +0000 UTC m=+37.584963155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert") pod "ingress-canary-47c8w" (UID: "ba8f16cc-755b-4dc9-a1b5-3c33e7149c77") : secret "canary-serving-cert" not found
May 28 23:02:42.486708 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:42.486684    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:02:42.486708 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:42.486694    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:02:42.486960 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:42.486685    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:42.490619 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:42.490592    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-65lhq\""
May 28 23:02:42.490619 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:42.490607    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 28 23:02:42.490816 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:42.490625    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 28 23:02:42.490816 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:42.490685    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 28 23:02:42.490816 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:42.490630    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 28 23:02:42.490816 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:42.490725    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6z5pd\""
May 28 23:02:42.722529 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:42.722495    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvj99" event={"ID":"321b5d1e-083e-45f0-99c7-066f9ec958d2","Type":"ContainerStarted","Data":"2c05e48e5a6d8a98b12fd42fdf3cf2165020119c34845eba92db92c561e42716"}
May 28 23:02:43.728200 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:43.728157    2577 generic.go:358] "Generic (PLEG): container finished" podID="321b5d1e-083e-45f0-99c7-066f9ec958d2" containerID="2c05e48e5a6d8a98b12fd42fdf3cf2165020119c34845eba92db92c561e42716" exitCode=0
May 28 23:02:43.728569 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:43.728219    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvj99" event={"ID":"321b5d1e-083e-45f0-99c7-066f9ec958d2","Type":"ContainerDied","Data":"2c05e48e5a6d8a98b12fd42fdf3cf2165020119c34845eba92db92c561e42716"}
May 28 23:02:44.404020 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:44.403983    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:44.404195 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:44.404034    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert\") pod \"ingress-canary-47c8w\" (UID: \"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77\") " pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:02:44.404195 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:44.404125    2577 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 23:02:44.404195 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:44.404129    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 23:02:44.404195 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:44.404175    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert podName:ba8f16cc-755b-4dc9-a1b5-3c33e7149c77 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:48.404161933 +0000 UTC m=+41.597593162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert") pod "ingress-canary-47c8w" (UID: "ba8f16cc-755b-4dc9-a1b5-3c33e7149c77") : secret "canary-serving-cert" not found
May 28 23:02:44.404195 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:44.404187    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls podName:0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b nodeName:}" failed. No retries permitted until 2026-05-28 23:02:48.404181838 +0000 UTC m=+41.597613068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls") pod "dns-default-n77fp" (UID: "0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b") : secret "dns-default-metrics-tls" not found
May 28 23:02:44.733111 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:44.733026    2577 generic.go:358] "Generic (PLEG): container finished" podID="321b5d1e-083e-45f0-99c7-066f9ec958d2" containerID="cfabb99a3d49aa95128aec8f7b69a851ebbfa1bfbe99a0ab715508080e97e198" exitCode=0
May 28 23:02:44.733111 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:44.733074    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvj99" event={"ID":"321b5d1e-083e-45f0-99c7-066f9ec958d2","Type":"ContainerDied","Data":"cfabb99a3d49aa95128aec8f7b69a851ebbfa1bfbe99a0ab715508080e97e198"}
May 28 23:02:45.737669 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:45.737635    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvj99" event={"ID":"321b5d1e-083e-45f0-99c7-066f9ec958d2","Type":"ContainerStarted","Data":"69bbaad0d96917af239db15b5b665f0907b9c46f6842c7d7272af547f9223a4a"}
May 28 23:02:45.765777 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:45.765719    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zvj99" podStartSLOduration=4.971356183 podStartE2EDuration="38.76570386s" podCreationTimestamp="2026-05-28 23:02:07 +0000 UTC" firstStartedPulling="2026-05-28 23:02:08.721897463 +0000 UTC m=+1.915328695" lastFinishedPulling="2026-05-28 23:02:42.516245127 +0000 UTC m=+35.709676372" observedRunningTime="2026-05-28 23:02:45.764144162 +0000 UTC m=+38.957575413" watchObservedRunningTime="2026-05-28 23:02:45.76570386 +0000 UTC m=+38.959135112"
May 28 23:02:48.429172 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:48.429128    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:48.429548 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:48.429183    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert\") pod \"ingress-canary-47c8w\" (UID: \"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77\") " pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:02:48.429548 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:48.429273    2577 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 23:02:48.429548 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:48.429278    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 23:02:48.429548 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:48.429328    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert podName:ba8f16cc-755b-4dc9-a1b5-3c33e7149c77 nodeName:}" failed. No retries permitted until 2026-05-28 23:02:56.429313075 +0000 UTC m=+49.622744305 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert") pod "ingress-canary-47c8w" (UID: "ba8f16cc-755b-4dc9-a1b5-3c33e7149c77") : secret "canary-serving-cert" not found
May 28 23:02:48.429548 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:48.429341    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls podName:0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b nodeName:}" failed. No retries permitted until 2026-05-28 23:02:56.429334822 +0000 UTC m=+49.622766052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls") pod "dns-default-n77fp" (UID: "0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b") : secret "dns-default-metrics-tls" not found
May 28 23:02:54.977830 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:54.977782    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret\") pod \"global-pull-secret-syncer-xtwjw\" (UID: \"1f610866-a2a5-4395-bcad-928e420c46bb\") " pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:54.981446 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:54.981420    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f610866-a2a5-4395-bcad-928e420c46bb-original-pull-secret\") pod \"global-pull-secret-syncer-xtwjw\" (UID: \"1f610866-a2a5-4395-bcad-928e420c46bb\") " pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:55.103281 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:55.103243    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtwjw"
May 28 23:02:55.291800 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:55.291772    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xtwjw"]
May 28 23:02:55.756002 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:55.755949    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xtwjw" event={"ID":"1f610866-a2a5-4395-bcad-928e420c46bb","Type":"ContainerStarted","Data":"524d1f4ab7c7299de9df21b5a6a4678bf2fccb775a084eca9e4c43125be60b2f"}
May 28 23:02:56.489818 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:56.489778    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:02:56.490306 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:56.489836    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert\") pod \"ingress-canary-47c8w\" (UID: \"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77\") " pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:02:56.490306 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:56.489959    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 23:02:56.490306 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:56.489964    2577 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 23:02:56.490306 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:56.490057    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls podName:0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b nodeName:}" failed. No retries permitted until 2026-05-28 23:03:12.490034524 +0000 UTC m=+65.683465758 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls") pod "dns-default-n77fp" (UID: "0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b") : secret "dns-default-metrics-tls" not found
May 28 23:02:56.490306 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:02:56.490120    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert podName:ba8f16cc-755b-4dc9-a1b5-3c33e7149c77 nodeName:}" failed. No retries permitted until 2026-05-28 23:03:12.490105341 +0000 UTC m=+65.683536574 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert") pod "ingress-canary-47c8w" (UID: "ba8f16cc-755b-4dc9-a1b5-3c33e7149c77") : secret "canary-serving-cert" not found
May 28 23:02:59.765988 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:59.765953    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xtwjw" event={"ID":"1f610866-a2a5-4395-bcad-928e420c46bb","Type":"ContainerStarted","Data":"acd03487f021db32178062adc4f5b342d90ef5547191f1fb73a10e4fb4efc060"}
May 28 23:02:59.783052 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:02:59.783006    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xtwjw" podStartSLOduration=33.054951114 podStartE2EDuration="36.782992957s" podCreationTimestamp="2026-05-28 23:02:23 +0000 UTC" firstStartedPulling="2026-05-28 23:02:55.297096633 +0000 UTC m=+48.490527862" lastFinishedPulling="2026-05-28 23:02:59.025138471 +0000 UTC m=+52.218569705" observedRunningTime="2026-05-28 23:02:59.782137581 +0000 UTC m=+52.975568829" watchObservedRunningTime="2026-05-28 23:02:59.782992957 +0000 UTC m=+52.976424202"
May 28 23:03:06.716708 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:06.716674    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w59bt"
May 28 23:03:12.102992 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:12.102931    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs\") pod \"network-metrics-daemon-mwvzm\" (UID: \"0a871ef7-90af-4a70-96ae-7df74d7ca072\") " pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:03:12.105936 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:12.105914    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 28 23:03:12.113390 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:12.113366    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 28 23:03:12.113491 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:12.113441    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs podName:0a871ef7-90af-4a70-96ae-7df74d7ca072 nodeName:}" failed. No retries permitted until 2026-05-28 23:04:16.11341761 +0000 UTC m=+129.306848856 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs") pod "network-metrics-daemon-mwvzm" (UID: "0a871ef7-90af-4a70-96ae-7df74d7ca072") : secret "metrics-daemon-secret" not found
May 28 23:03:12.203958 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:12.203910    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-74ggn\" (UniqueName: \"kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn\") pod \"network-check-target-2kktk\" (UID: \"9c10eeed-3826-4345-9e9b-6b8d25111763\") " pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:03:12.207203 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:12.207157    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 28 23:03:12.217632 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:12.217609    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 28 23:03:12.227555 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:12.227521    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ggn\" (UniqueName: \"kubernetes.io/projected/9c10eeed-3826-4345-9e9b-6b8d25111763-kube-api-access-74ggn\") pod \"network-check-target-2kktk\" (UID: \"9c10eeed-3826-4345-9e9b-6b8d25111763\") " pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:03:12.500123 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:12.500097    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6z5pd\""
May 28 23:03:12.506911 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:12.506886    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:03:12.506980 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:12.506944    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert\") pod \"ingress-canary-47c8w\" (UID: \"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77\") " pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:03:12.506980 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:12.506957    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 23:03:12.507043 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:12.507008    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls podName:0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b nodeName:}" failed. No retries permitted until 2026-05-28 23:03:44.506991746 +0000 UTC m=+97.700422976 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls") pod "dns-default-n77fp" (UID: "0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b") : secret "dns-default-metrics-tls" not found
May 28 23:03:12.507043 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:12.507032    2577 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 23:03:12.507118 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:12.507071    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert podName:ba8f16cc-755b-4dc9-a1b5-3c33e7149c77 nodeName:}" failed. No retries permitted until 2026-05-28 23:03:44.507056283 +0000 UTC m=+97.700487512 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert") pod "ingress-canary-47c8w" (UID: "ba8f16cc-755b-4dc9-a1b5-3c33e7149c77") : secret "canary-serving-cert" not found
May 28 23:03:12.507118 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:12.507112    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:03:12.644186 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:12.644152    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2kktk"]
May 28 23:03:12.647965 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:03:12.647938    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c10eeed_3826_4345_9e9b_6b8d25111763.slice/crio-674b3724eb8280fea22e5b0b14ecac2afc66896d003c064545ce817ba1cd0094 WatchSource:0}: Error finding container 674b3724eb8280fea22e5b0b14ecac2afc66896d003c064545ce817ba1cd0094: Status 404 returned error can't find the container with id 674b3724eb8280fea22e5b0b14ecac2afc66896d003c064545ce817ba1cd0094
May 28 23:03:12.792005 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:12.791911    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2kktk" event={"ID":"9c10eeed-3826-4345-9e9b-6b8d25111763","Type":"ContainerStarted","Data":"674b3724eb8280fea22e5b0b14ecac2afc66896d003c064545ce817ba1cd0094"}
May 28 23:03:15.798939 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:15.798894    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2kktk" event={"ID":"9c10eeed-3826-4345-9e9b-6b8d25111763","Type":"ContainerStarted","Data":"af5fcebb24af779fc351a01186ff26282c522d04e9af61255ce083bc124b04ec"}
May 28 23:03:15.799333 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:15.799040    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:03:42.347679 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.347621    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2kktk" podStartSLOduration=92.579595762 podStartE2EDuration="1m35.347606328s" podCreationTimestamp="2026-05-28 23:02:07 +0000 UTC" firstStartedPulling="2026-05-28 23:03:12.649910876 +0000 UTC m=+65.843342106" lastFinishedPulling="2026-05-28 23:03:15.417921424 +0000 UTC m=+68.611352672" observedRunningTime="2026-05-28 23:03:15.815248949 +0000 UTC m=+69.008680202" watchObservedRunningTime="2026-05-28 23:03:42.347606328 +0000 UTC m=+95.541037617"
May 28 23:03:42.348269 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.348253    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"]
May 28 23:03:42.350590 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.350575    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:03:42.352924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.352900    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
May 28 23:03:42.353070 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.352990    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 28 23:03:42.353070 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.353001    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6xtf6\""
May 28 23:03:42.353826 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.353811    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 28 23:03:42.355026 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.355009    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
May 28 23:03:42.358389 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.358365    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"]
May 28 23:03:42.410328 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.410298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9vkd\" (UniqueName: \"kubernetes.io/projected/9d86f162-8349-42e1-abd1-aef9e7c328a4-kube-api-access-f9vkd\") pod \"cluster-monitoring-operator-6c75959455-xk7gh\" (UID: \"9d86f162-8349-42e1-abd1-aef9e7c328a4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:03:42.410495 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.410348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9d86f162-8349-42e1-abd1-aef9e7c328a4-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-xk7gh\" (UID: \"9d86f162-8349-42e1-abd1-aef9e7c328a4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:03:42.410495 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.410373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xk7gh\" (UID: \"9d86f162-8349-42e1-abd1-aef9e7c328a4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:03:42.511582 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.511553    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f9vkd\" (UniqueName: \"kubernetes.io/projected/9d86f162-8349-42e1-abd1-aef9e7c328a4-kube-api-access-f9vkd\") pod \"cluster-monitoring-operator-6c75959455-xk7gh\" (UID: \"9d86f162-8349-42e1-abd1-aef9e7c328a4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:03:42.511732 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.511591    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9d86f162-8349-42e1-abd1-aef9e7c328a4-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-xk7gh\" (UID: \"9d86f162-8349-42e1-abd1-aef9e7c328a4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:03:42.511732 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.511609    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xk7gh\" (UID: \"9d86f162-8349-42e1-abd1-aef9e7c328a4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:03:42.511732 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:42.511704    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 23:03:42.511854 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:42.511762    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls podName:9d86f162-8349-42e1-abd1-aef9e7c328a4 nodeName:}" failed. No retries permitted until 2026-05-28 23:03:43.011748569 +0000 UTC m=+96.205179800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xk7gh" (UID: "9d86f162-8349-42e1-abd1-aef9e7c328a4") : secret "cluster-monitoring-operator-tls" not found
May 28 23:03:42.512377 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.512357    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9d86f162-8349-42e1-abd1-aef9e7c328a4-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-xk7gh\" (UID: \"9d86f162-8349-42e1-abd1-aef9e7c328a4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:03:42.521875 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:42.521849    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9vkd\" (UniqueName: \"kubernetes.io/projected/9d86f162-8349-42e1-abd1-aef9e7c328a4-kube-api-access-f9vkd\") pod \"cluster-monitoring-operator-6c75959455-xk7gh\" (UID: \"9d86f162-8349-42e1-abd1-aef9e7c328a4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:03:43.014538 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:43.014507    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xk7gh\" (UID: \"9d86f162-8349-42e1-abd1-aef9e7c328a4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:03:43.014707 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:43.014606    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 23:03:43.014707 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:43.014659    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls podName:9d86f162-8349-42e1-abd1-aef9e7c328a4 nodeName:}" failed. No retries permitted until 2026-05-28 23:03:44.014644918 +0000 UTC m=+97.208076150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xk7gh" (UID: "9d86f162-8349-42e1-abd1-aef9e7c328a4") : secret "cluster-monitoring-operator-tls" not found
May 28 23:03:44.021141 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:44.021090    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xk7gh\" (UID: \"9d86f162-8349-42e1-abd1-aef9e7c328a4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:03:44.021524 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:44.021257    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 23:03:44.021524 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:44.021342    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls podName:9d86f162-8349-42e1-abd1-aef9e7c328a4 nodeName:}" failed. No retries permitted until 2026-05-28 23:03:46.021325557 +0000 UTC m=+99.214756791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xk7gh" (UID: "9d86f162-8349-42e1-abd1-aef9e7c328a4") : secret "cluster-monitoring-operator-tls" not found
May 28 23:03:44.524631 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:44.524584    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:03:44.524631 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:44.524646    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert\") pod \"ingress-canary-47c8w\" (UID: \"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77\") " pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:03:44.524848 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:44.524742    2577 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 23:03:44.524848 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:44.524754    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 23:03:44.524848 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:44.524823    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert podName:ba8f16cc-755b-4dc9-a1b5-3c33e7149c77 nodeName:}" failed. No retries permitted until 2026-05-28 23:04:48.524808754 +0000 UTC m=+161.718239984 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert") pod "ingress-canary-47c8w" (UID: "ba8f16cc-755b-4dc9-a1b5-3c33e7149c77") : secret "canary-serving-cert" not found
May 28 23:03:44.524848 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:44.524836    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls podName:0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b nodeName:}" failed. No retries permitted until 2026-05-28 23:04:48.524830736 +0000 UTC m=+161.718261966 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls") pod "dns-default-n77fp" (UID: "0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b") : secret "dns-default-metrics-tls" not found
May 28 23:03:46.036387 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.036350    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xk7gh\" (UID: \"9d86f162-8349-42e1-abd1-aef9e7c328a4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:03:46.036774 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:46.036468    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 23:03:46.036774 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:46.036520    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls podName:9d86f162-8349-42e1-abd1-aef9e7c328a4 nodeName:}" failed. No retries permitted until 2026-05-28 23:03:50.036506579 +0000 UTC m=+103.229937814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xk7gh" (UID: "9d86f162-8349-42e1-abd1-aef9e7c328a4") : secret "cluster-monitoring-operator-tls" not found
May 28 23:03:46.421283 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.421206    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-zzfbm"]
May 28 23:03:46.446801 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.446776    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-zzfbm"]
May 28 23:03:46.446963 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.446904    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-zzfbm"
May 28 23:03:46.449580 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.449559    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 28 23:03:46.450520 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.450503    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 28 23:03:46.450591 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.450570    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-45b85\""
May 28 23:03:46.521624 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.521595    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"]
May 28 23:03:46.538213 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.538192    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"]
May 28 23:03:46.538319 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.538293    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"
May 28 23:03:46.540631 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.540596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8lb4\" (UniqueName: \"kubernetes.io/projected/e9d3fa19-567d-4f02-bd2c-1dd014be825e-kube-api-access-n8lb4\") pod \"volume-data-source-validator-5cf8c89d64-zzfbm\" (UID: \"e9d3fa19-567d-4f02-bd2c-1dd014be825e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-zzfbm"
May 28 23:03:46.540882 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.540847    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 28 23:03:46.540980 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.540955    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 28 23:03:46.541091 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.541073    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 28 23:03:46.541160 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.541091    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hz76d\""
May 28 23:03:46.641159 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.641125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x4gsd\" (UID: \"c93652d2-0a39-40de-ae3c-e31fa167e356\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"
May 28 23:03:46.641159 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.641162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nzc9\" (UniqueName: \"kubernetes.io/projected/c93652d2-0a39-40de-ae3c-e31fa167e356-kube-api-access-9nzc9\") pod \"cluster-samples-operator-697587599-x4gsd\" (UID: \"c93652d2-0a39-40de-ae3c-e31fa167e356\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"
May 28 23:03:46.641344 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.641208    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n8lb4\" (UniqueName: \"kubernetes.io/projected/e9d3fa19-567d-4f02-bd2c-1dd014be825e-kube-api-access-n8lb4\") pod \"volume-data-source-validator-5cf8c89d64-zzfbm\" (UID: \"e9d3fa19-567d-4f02-bd2c-1dd014be825e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-zzfbm"
May 28 23:03:46.650392 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.650370    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8lb4\" (UniqueName: \"kubernetes.io/projected/e9d3fa19-567d-4f02-bd2c-1dd014be825e-kube-api-access-n8lb4\") pod \"volume-data-source-validator-5cf8c89d64-zzfbm\" (UID: \"e9d3fa19-567d-4f02-bd2c-1dd014be825e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-zzfbm"
May 28 23:03:46.742023 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.741999    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x4gsd\" (UID: \"c93652d2-0a39-40de-ae3c-e31fa167e356\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"
May 28 23:03:46.742148 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.742033    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzc9\" (UniqueName: \"kubernetes.io/projected/c93652d2-0a39-40de-ae3c-e31fa167e356-kube-api-access-9nzc9\") pod \"cluster-samples-operator-697587599-x4gsd\" (UID: \"c93652d2-0a39-40de-ae3c-e31fa167e356\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"
May 28 23:03:46.742186 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:46.742151    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 23:03:46.742228 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:46.742217    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls podName:c93652d2-0a39-40de-ae3c-e31fa167e356 nodeName:}" failed. No retries permitted until 2026-05-28 23:03:47.242198973 +0000 UTC m=+100.435630209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls") pod "cluster-samples-operator-697587599-x4gsd" (UID: "c93652d2-0a39-40de-ae3c-e31fa167e356") : secret "samples-operator-tls" not found
May 28 23:03:46.755831 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.755806    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-zzfbm"
May 28 23:03:46.757074 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.757056    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzc9\" (UniqueName: \"kubernetes.io/projected/c93652d2-0a39-40de-ae3c-e31fa167e356-kube-api-access-9nzc9\") pod \"cluster-samples-operator-697587599-x4gsd\" (UID: \"c93652d2-0a39-40de-ae3c-e31fa167e356\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"
May 28 23:03:46.803778 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.803746    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2kktk"
May 28 23:03:46.878711 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:46.878679    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-zzfbm"]
May 28 23:03:46.881561 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:03:46.881531    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d3fa19_567d_4f02_bd2c_1dd014be825e.slice/crio-d2040d957c6ec2435f9f5ae495b7ac14eda545f24cf3a97039a96ee0aac6e163 WatchSource:0}: Error finding container d2040d957c6ec2435f9f5ae495b7ac14eda545f24cf3a97039a96ee0aac6e163: Status 404 returned error can't find the container with id d2040d957c6ec2435f9f5ae495b7ac14eda545f24cf3a97039a96ee0aac6e163
May 28 23:03:47.246295 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:47.246268    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x4gsd\" (UID: \"c93652d2-0a39-40de-ae3c-e31fa167e356\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"
May 28 23:03:47.246671 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:47.246378    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 23:03:47.246671 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:47.246435    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls podName:c93652d2-0a39-40de-ae3c-e31fa167e356 nodeName:}" failed. No retries permitted until 2026-05-28 23:03:48.246418224 +0000 UTC m=+101.439849470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls") pod "cluster-samples-operator-697587599-x4gsd" (UID: "c93652d2-0a39-40de-ae3c-e31fa167e356") : secret "samples-operator-tls" not found
May 28 23:03:47.856764 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:47.856718    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-zzfbm" event={"ID":"e9d3fa19-567d-4f02-bd2c-1dd014be825e","Type":"ContainerStarted","Data":"d2040d957c6ec2435f9f5ae495b7ac14eda545f24cf3a97039a96ee0aac6e163"}
May 28 23:03:48.253953 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.253918    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x4gsd\" (UID: \"c93652d2-0a39-40de-ae3c-e31fa167e356\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"
May 28 23:03:48.254337 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:48.254104    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 23:03:48.254337 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:48.254189    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls podName:c93652d2-0a39-40de-ae3c-e31fa167e356 nodeName:}" failed. No retries permitted until 2026-05-28 23:03:50.254166318 +0000 UTC m=+103.447597554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls") pod "cluster-samples-operator-697587599-x4gsd" (UID: "c93652d2-0a39-40de-ae3c-e31fa167e356") : secret "samples-operator-tls" not found
May 28 23:03:48.420339 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.420301    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-nj29z"]
May 28 23:03:48.423061 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.423038    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:48.425522 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.425491    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 28 23:03:48.425522 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.425504    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 28 23:03:48.425712 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.425610    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-54pn9\""
May 28 23:03:48.426414 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.426394    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 28 23:03:48.426541 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.426396    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 28 23:03:48.430257 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.430235    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 28 23:03:48.431982 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.431963    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-nj29z"]
May 28 23:03:48.456257 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.456235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edcd544f-4680-4190-a9b0-453053515b7a-trusted-ca\") pod \"console-operator-76dff48d59-nj29z\" (UID: \"edcd544f-4680-4190-a9b0-453053515b7a\") " pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:48.456343 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.456266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edcd544f-4680-4190-a9b0-453053515b7a-config\") pod \"console-operator-76dff48d59-nj29z\" (UID: \"edcd544f-4680-4190-a9b0-453053515b7a\") " pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:48.456343 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.456297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edcd544f-4680-4190-a9b0-453053515b7a-serving-cert\") pod \"console-operator-76dff48d59-nj29z\" (UID: \"edcd544f-4680-4190-a9b0-453053515b7a\") " pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:48.456487 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.456382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvb9t\" (UniqueName: \"kubernetes.io/projected/edcd544f-4680-4190-a9b0-453053515b7a-kube-api-access-lvb9t\") pod \"console-operator-76dff48d59-nj29z\" (UID: \"edcd544f-4680-4190-a9b0-453053515b7a\") " pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:48.556925 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.556838    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edcd544f-4680-4190-a9b0-453053515b7a-serving-cert\") pod \"console-operator-76dff48d59-nj29z\" (UID: \"edcd544f-4680-4190-a9b0-453053515b7a\") " pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:48.556925 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.556907    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lvb9t\" (UniqueName: \"kubernetes.io/projected/edcd544f-4680-4190-a9b0-453053515b7a-kube-api-access-lvb9t\") pod \"console-operator-76dff48d59-nj29z\" (UID: \"edcd544f-4680-4190-a9b0-453053515b7a\") " pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:48.557123 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.556971    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edcd544f-4680-4190-a9b0-453053515b7a-trusted-ca\") pod \"console-operator-76dff48d59-nj29z\" (UID: \"edcd544f-4680-4190-a9b0-453053515b7a\") " pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:48.557123 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.557000    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edcd544f-4680-4190-a9b0-453053515b7a-config\") pod \"console-operator-76dff48d59-nj29z\" (UID: \"edcd544f-4680-4190-a9b0-453053515b7a\") " pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:48.557602 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.557578    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/edcd544f-4680-4190-a9b0-453053515b7a-config\") pod \"console-operator-76dff48d59-nj29z\" (UID: \"edcd544f-4680-4190-a9b0-453053515b7a\") " pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:48.557803 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.557780    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edcd544f-4680-4190-a9b0-453053515b7a-trusted-ca\") pod \"console-operator-76dff48d59-nj29z\" (UID: \"edcd544f-4680-4190-a9b0-453053515b7a\") " pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:48.559407 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.559384    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/edcd544f-4680-4190-a9b0-453053515b7a-serving-cert\") pod \"console-operator-76dff48d59-nj29z\" (UID: \"edcd544f-4680-4190-a9b0-453053515b7a\") " pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:48.560162 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.560142    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rhwv2_8856d7aa-86ca-4ef4-9d92-ceac3dd18292/dns-node-resolver/0.log"
May 28 23:03:48.565600 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.565573    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvb9t\" (UniqueName: \"kubernetes.io/projected/edcd544f-4680-4190-a9b0-453053515b7a-kube-api-access-lvb9t\") pod \"console-operator-76dff48d59-nj29z\" (UID: \"edcd544f-4680-4190-a9b0-453053515b7a\") " pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:48.732749 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.732707    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:48.852484 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.852417    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-nj29z"]
May 28 23:03:48.855130 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:03:48.855102    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcd544f_4680_4190_a9b0_453053515b7a.slice/crio-1dea0422e7ea6f7c3e2d02e1f0055c8db05827305a2e1fc4993d309517bb9eb0 WatchSource:0}: Error finding container 1dea0422e7ea6f7c3e2d02e1f0055c8db05827305a2e1fc4993d309517bb9eb0: Status 404 returned error can't find the container with id 1dea0422e7ea6f7c3e2d02e1f0055c8db05827305a2e1fc4993d309517bb9eb0
May 28 23:03:48.859346 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.859320    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-zzfbm" event={"ID":"e9d3fa19-567d-4f02-bd2c-1dd014be825e","Type":"ContainerStarted","Data":"b4ab0f4af6e5ac178a7ec946614d04e5cc8a091fe848d6abf6ed5b15b521abc8"}
May 28 23:03:48.860320 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:48.860299    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-nj29z" event={"ID":"edcd544f-4680-4190-a9b0-453053515b7a","Type":"ContainerStarted","Data":"1dea0422e7ea6f7c3e2d02e1f0055c8db05827305a2e1fc4993d309517bb9eb0"}
May 28 23:03:49.560667 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:49.560641    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cm8m8_8353cc62-2b97-4f45-8a3d-b871840ee8e8/node-ca/0.log"
May 28 23:03:50.068320 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:50.068275    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xk7gh\" (UID: \"9d86f162-8349-42e1-abd1-aef9e7c328a4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:03:50.068494 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:50.068445    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 23:03:50.068535 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:50.068520    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls podName:9d86f162-8349-42e1-abd1-aef9e7c328a4 nodeName:}" failed. No retries permitted until 2026-05-28 23:03:58.06850271 +0000 UTC m=+111.261933939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xk7gh" (UID: "9d86f162-8349-42e1-abd1-aef9e7c328a4") : secret "cluster-monitoring-operator-tls" not found
May 28 23:03:50.270408 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:50.270367    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x4gsd\" (UID: \"c93652d2-0a39-40de-ae3c-e31fa167e356\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"
May 28 23:03:50.270576 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:50.270489    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 23:03:50.270576 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:50.270548    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls podName:c93652d2-0a39-40de-ae3c-e31fa167e356 nodeName:}" failed. No retries permitted until 2026-05-28 23:03:54.270529725 +0000 UTC m=+107.463960972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls") pod "cluster-samples-operator-697587599-x4gsd" (UID: "c93652d2-0a39-40de-ae3c-e31fa167e356") : secret "samples-operator-tls" not found
May 28 23:03:51.867921 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:51.867891    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/0.log"
May 28 23:03:51.868274 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:51.867932    2577 generic.go:358] "Generic (PLEG): container finished" podID="edcd544f-4680-4190-a9b0-453053515b7a" containerID="df98690cdaeb1483bbefed8f4b0eee9c5e75c77052245b916d35ce03c0e879ef" exitCode=255
May 28 23:03:51.868274 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:51.867964    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-nj29z" event={"ID":"edcd544f-4680-4190-a9b0-453053515b7a","Type":"ContainerDied","Data":"df98690cdaeb1483bbefed8f4b0eee9c5e75c77052245b916d35ce03c0e879ef"}
May 28 23:03:51.868274 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:51.868214    2577 scope.go:117] "RemoveContainer" containerID="df98690cdaeb1483bbefed8f4b0eee9c5e75c77052245b916d35ce03c0e879ef"
May 28 23:03:51.884122 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:51.884075    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-zzfbm" podStartSLOduration=4.323259456 podStartE2EDuration="5.884056465s" podCreationTimestamp="2026-05-28 23:03:46 +0000 UTC" firstStartedPulling="2026-05-28 23:03:46.883542503 +0000 UTC m=+100.076973752" lastFinishedPulling="2026-05-28 23:03:48.444339528 +0000 UTC m=+101.637770761" observedRunningTime="2026-05-28 23:03:48.874627658 +0000 UTC m=+102.068058921" watchObservedRunningTime="2026-05-28 23:03:51.884056465 +0000 UTC m=+105.077487722"
May 28 23:03:52.871047 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:52.871019    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/1.log"
May 28 23:03:52.871457 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:52.871393    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/0.log"
May 28 23:03:52.871457 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:52.871429    2577 generic.go:358] "Generic (PLEG): container finished" podID="edcd544f-4680-4190-a9b0-453053515b7a" containerID="dbc52b0aec12f732c2d704a635db2a8ada450e80cb95c52c9794267f19887df5" exitCode=255
May 28 23:03:52.871526 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:52.871460    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-nj29z" event={"ID":"edcd544f-4680-4190-a9b0-453053515b7a","Type":"ContainerDied","Data":"dbc52b0aec12f732c2d704a635db2a8ada450e80cb95c52c9794267f19887df5"}
May 28 23:03:52.871526 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:52.871493    2577 scope.go:117] "RemoveContainer" containerID="df98690cdaeb1483bbefed8f4b0eee9c5e75c77052245b916d35ce03c0e879ef"
May 28 23:03:52.871744 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:52.871728    2577 scope.go:117] "RemoveContainer" containerID="dbc52b0aec12f732c2d704a635db2a8ada450e80cb95c52c9794267f19887df5"
May 28 23:03:52.871972 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:52.871954    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-nj29z_openshift-console-operator(edcd544f-4680-4190-a9b0-453053515b7a)\"" pod="openshift-console-operator/console-operator-76dff48d59-nj29z" podUID="edcd544f-4680-4190-a9b0-453053515b7a"
May 28 23:03:53.875118 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:53.875091    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/1.log"
May 28 23:03:53.875486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:53.875408    2577 scope.go:117] "RemoveContainer" containerID="dbc52b0aec12f732c2d704a635db2a8ada450e80cb95c52c9794267f19887df5"
May 28 23:03:53.875587 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:53.875569    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-nj29z_openshift-console-operator(edcd544f-4680-4190-a9b0-453053515b7a)\"" pod="openshift-console-operator/console-operator-76dff48d59-nj29z" podUID="edcd544f-4680-4190-a9b0-453053515b7a"
May 28 23:03:54.299064 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:54.299013    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x4gsd\" (UID: \"c93652d2-0a39-40de-ae3c-e31fa167e356\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"
May 28 23:03:54.299235 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:54.299168    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 23:03:54.299275 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:54.299248    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls podName:c93652d2-0a39-40de-ae3c-e31fa167e356 nodeName:}" failed. No retries permitted until 2026-05-28 23:04:02.299232083 +0000 UTC m=+115.492663312 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls") pod "cluster-samples-operator-697587599-x4gsd" (UID: "c93652d2-0a39-40de-ae3c-e31fa167e356") : secret "samples-operator-tls" not found
May 28 23:03:54.452059 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:54.452025    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-vfhc7"]
May 28 23:03:54.454738 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:54.454723    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-vfhc7"
May 28 23:03:54.457113 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:54.457093    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8rz7v\""
May 28 23:03:54.462048 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:54.462026    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-vfhc7"]
May 28 23:03:54.500313 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:54.500278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbmqc\" (UniqueName: \"kubernetes.io/projected/e6564129-6c7b-4789-aca5-0ce1eb814173-kube-api-access-qbmqc\") pod \"network-check-source-7f44b6dd87-vfhc7\" (UID: \"e6564129-6c7b-4789-aca5-0ce1eb814173\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-vfhc7"
May 28 23:03:54.601600 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:54.601508    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qbmqc\" (UniqueName: \"kubernetes.io/projected/e6564129-6c7b-4789-aca5-0ce1eb814173-kube-api-access-qbmqc\") pod \"network-check-source-7f44b6dd87-vfhc7\" (UID: \"e6564129-6c7b-4789-aca5-0ce1eb814173\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-vfhc7"
May 28 23:03:54.610285 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:54.610262    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbmqc\" (UniqueName: \"kubernetes.io/projected/e6564129-6c7b-4789-aca5-0ce1eb814173-kube-api-access-qbmqc\") pod \"network-check-source-7f44b6dd87-vfhc7\" (UID: \"e6564129-6c7b-4789-aca5-0ce1eb814173\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-vfhc7"
May 28 23:03:54.764310 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:54.764280    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-vfhc7"
May 28 23:03:54.883362 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:54.883287    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-vfhc7"]
May 28 23:03:54.886124 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:03:54.886095    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6564129_6c7b_4789_aca5_0ce1eb814173.slice/crio-1a39645ce9f0b31a0988b8fb707401b81a7e58c79d5f5d021f0feb019198f9e1 WatchSource:0}: Error finding container 1a39645ce9f0b31a0988b8fb707401b81a7e58c79d5f5d021f0feb019198f9e1: Status 404 returned error can't find the container with id 1a39645ce9f0b31a0988b8fb707401b81a7e58c79d5f5d021f0feb019198f9e1
May 28 23:03:55.657894 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:55.657850    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-p2l28"]
May 28 23:03:55.660858 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:55.660832    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-p2l28"
May 28 23:03:55.667423 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:55.667401    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-n2599\""
May 28 23:03:55.667894 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:55.667855    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
May 28 23:03:55.668209 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:55.668196    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
May 28 23:03:55.671819 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:55.671797    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-p2l28"]
May 28 23:03:55.710777 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:55.710750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqwkt\" (UniqueName: \"kubernetes.io/projected/175d6678-c45e-4aed-a2cb-d460d6d578cf-kube-api-access-cqwkt\") pod \"migrator-b985f5d89-p2l28\" (UID: \"175d6678-c45e-4aed-a2cb-d460d6d578cf\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-p2l28"
May 28 23:03:55.811484 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:55.811446    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cqwkt\" (UniqueName: \"kubernetes.io/projected/175d6678-c45e-4aed-a2cb-d460d6d578cf-kube-api-access-cqwkt\") pod \"migrator-b985f5d89-p2l28\" (UID: \"175d6678-c45e-4aed-a2cb-d460d6d578cf\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-p2l28"
May 28 23:03:55.824682 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:55.824654    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqwkt\" (UniqueName: \"kubernetes.io/projected/175d6678-c45e-4aed-a2cb-d460d6d578cf-kube-api-access-cqwkt\") pod \"migrator-b985f5d89-p2l28\" (UID: \"175d6678-c45e-4aed-a2cb-d460d6d578cf\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-p2l28"
May 28 23:03:55.880797 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:55.880762    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-vfhc7" event={"ID":"e6564129-6c7b-4789-aca5-0ce1eb814173","Type":"ContainerStarted","Data":"d9872a0a17f40ccb8dfa72d5ac062f08c3df52d09ed21e5f040d288af7c36bc4"}
May 28 23:03:55.880963 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:55.880807    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-vfhc7" event={"ID":"e6564129-6c7b-4789-aca5-0ce1eb814173","Type":"ContainerStarted","Data":"1a39645ce9f0b31a0988b8fb707401b81a7e58c79d5f5d021f0feb019198f9e1"}
May 28 23:03:55.897132 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:55.897090    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-vfhc7" podStartSLOduration=1.897075766 podStartE2EDuration="1.897075766s" podCreationTimestamp="2026-05-28 23:03:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:03:55.896744362 +0000 UTC m=+109.090175615" watchObservedRunningTime="2026-05-28 23:03:55.897075766 +0000 UTC m=+109.090507018"
May 28 23:03:55.970027 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:55.969948    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-p2l28"
May 28 23:03:56.090216 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:56.090179    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-p2l28"]
May 28 23:03:56.094279 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:03:56.094236    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175d6678_c45e_4aed_a2cb_d460d6d578cf.slice/crio-a765e310284ecd7bb59821c62107ae4c8e321ce0850fcf9c3aa3c18f08f2168e WatchSource:0}: Error finding container a765e310284ecd7bb59821c62107ae4c8e321ce0850fcf9c3aa3c18f08f2168e: Status 404 returned error can't find the container with id a765e310284ecd7bb59821c62107ae4c8e321ce0850fcf9c3aa3c18f08f2168e
May 28 23:03:56.884041 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:56.884003    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-p2l28" event={"ID":"175d6678-c45e-4aed-a2cb-d460d6d578cf","Type":"ContainerStarted","Data":"a765e310284ecd7bb59821c62107ae4c8e321ce0850fcf9c3aa3c18f08f2168e"}
May 28 23:03:57.888185 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:57.888148    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-p2l28" event={"ID":"175d6678-c45e-4aed-a2cb-d460d6d578cf","Type":"ContainerStarted","Data":"7e7a8d1b06053a592eba9a9b58c42c403f6c4406f4f333ad4bc99bb57d0678d9"}
May 28 23:03:57.888185 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:57.888189    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-p2l28" event={"ID":"175d6678-c45e-4aed-a2cb-d460d6d578cf","Type":"ContainerStarted","Data":"8855a4bab9d1ccbbc33f66972044c22cc8fddea28006615a115136a6066ca2a0"}
May 28 23:03:58.129282 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:58.129246    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xk7gh\" (UID: \"9d86f162-8349-42e1-abd1-aef9e7c328a4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:03:58.129443 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:58.129387    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 23:03:58.129483 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:58.129450    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls podName:9d86f162-8349-42e1-abd1-aef9e7c328a4 nodeName:}" failed. No retries permitted until 2026-05-28 23:04:14.129432168 +0000 UTC m=+127.322863401 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xk7gh" (UID: "9d86f162-8349-42e1-abd1-aef9e7c328a4") : secret "cluster-monitoring-operator-tls" not found
May 28 23:03:58.733275 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:58.733230    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:58.733275 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:58.733282    2577 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:03:58.733740 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:58.733723    2577 scope.go:117] "RemoveContainer" containerID="dbc52b0aec12f732c2d704a635db2a8ada450e80cb95c52c9794267f19887df5"
May 28 23:03:58.733966 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:03:58.733944    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-nj29z_openshift-console-operator(edcd544f-4680-4190-a9b0-453053515b7a)\"" pod="openshift-console-operator/console-operator-76dff48d59-nj29z" podUID="edcd544f-4680-4190-a9b0-453053515b7a"
May 28 23:03:58.964692 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:58.964649    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-p2l28" podStartSLOduration=2.828906023 podStartE2EDuration="3.964635331s" podCreationTimestamp="2026-05-28 23:03:55 +0000 UTC" firstStartedPulling="2026-05-28 23:03:56.096215649 +0000 UTC m=+109.289646880" lastFinishedPulling="2026-05-28 23:03:57.231944944 +0000 UTC m=+110.425376188" observedRunningTime="2026-05-28 23:03:57.903861673 +0000 UTC m=+111.097292925" watchObservedRunningTime="2026-05-28 23:03:58.964635331 +0000 UTC m=+112.158066605"
May 28 23:03:58.965297 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:58.965280    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-fsdnz"]
May 28 23:03:58.968235 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:58.968219    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-fsdnz"
May 28 23:03:58.970823 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:58.970802    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ddh94\""
May 28 23:03:58.970964 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:58.970922    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 28 23:03:58.971749 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:58.971731    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 28 23:03:58.971816 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:58.971791    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 28 23:03:58.971816 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:58.971801    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 28 23:03:58.975221 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:58.975036    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-fsdnz"]
May 28 23:03:59.035503 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:59.035462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c347fe6-bfc7-4d0a-8974-92bb5edc26f9-signing-cabundle\") pod \"service-ca-67d667c7d-fsdnz\" (UID: \"9c347fe6-bfc7-4d0a-8974-92bb5edc26f9\") " pod="openshift-service-ca/service-ca-67d667c7d-fsdnz"
May 28 23:03:59.035687 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:59.035515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9c347fe6-bfc7-4d0a-8974-92bb5edc26f9-signing-key\") pod \"service-ca-67d667c7d-fsdnz\" (UID: \"9c347fe6-bfc7-4d0a-8974-92bb5edc26f9\") " pod="openshift-service-ca/service-ca-67d667c7d-fsdnz"
May 28 23:03:59.035687 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:59.035552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxwq2\" (UniqueName: \"kubernetes.io/projected/9c347fe6-bfc7-4d0a-8974-92bb5edc26f9-kube-api-access-bxwq2\") pod \"service-ca-67d667c7d-fsdnz\" (UID: \"9c347fe6-bfc7-4d0a-8974-92bb5edc26f9\") " pod="openshift-service-ca/service-ca-67d667c7d-fsdnz"
May 28 23:03:59.136898 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:59.136846    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c347fe6-bfc7-4d0a-8974-92bb5edc26f9-signing-cabundle\") pod \"service-ca-67d667c7d-fsdnz\" (UID: \"9c347fe6-bfc7-4d0a-8974-92bb5edc26f9\") " pod="openshift-service-ca/service-ca-67d667c7d-fsdnz"
May 28 23:03:59.136898 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:59.136904    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9c347fe6-bfc7-4d0a-8974-92bb5edc26f9-signing-key\") pod \"service-ca-67d667c7d-fsdnz\" (UID: \"9c347fe6-bfc7-4d0a-8974-92bb5edc26f9\") " pod="openshift-service-ca/service-ca-67d667c7d-fsdnz"
May 28 23:03:59.137084 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:59.136929    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bxwq2\" (UniqueName: \"kubernetes.io/projected/9c347fe6-bfc7-4d0a-8974-92bb5edc26f9-kube-api-access-bxwq2\") pod \"service-ca-67d667c7d-fsdnz\" (UID: \"9c347fe6-bfc7-4d0a-8974-92bb5edc26f9\") " pod="openshift-service-ca/service-ca-67d667c7d-fsdnz"
May 28 23:03:59.137493 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:59.137468    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c347fe6-bfc7-4d0a-8974-92bb5edc26f9-signing-cabundle\") pod \"service-ca-67d667c7d-fsdnz\" (UID: \"9c347fe6-bfc7-4d0a-8974-92bb5edc26f9\") " pod="openshift-service-ca/service-ca-67d667c7d-fsdnz"
May 28 23:03:59.139450 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:59.139426    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9c347fe6-bfc7-4d0a-8974-92bb5edc26f9-signing-key\") pod \"service-ca-67d667c7d-fsdnz\" (UID: \"9c347fe6-bfc7-4d0a-8974-92bb5edc26f9\") " pod="openshift-service-ca/service-ca-67d667c7d-fsdnz"
May 28 23:03:59.144897 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:59.144861    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxwq2\" (UniqueName: \"kubernetes.io/projected/9c347fe6-bfc7-4d0a-8974-92bb5edc26f9-kube-api-access-bxwq2\") pod \"service-ca-67d667c7d-fsdnz\" (UID: \"9c347fe6-bfc7-4d0a-8974-92bb5edc26f9\") " pod="openshift-service-ca/service-ca-67d667c7d-fsdnz"
May 28 23:03:59.277041 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:59.277005    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-fsdnz"
May 28 23:03:59.397476 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:59.397445    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-fsdnz"]
May 28 23:03:59.402083 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:03:59.402048    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c347fe6_bfc7_4d0a_8974_92bb5edc26f9.slice/crio-613f00a6e745c3cfb68474cc7ddf01fb8fd9b539e68a4d0169266d683142e3ca WatchSource:0}: Error finding container 613f00a6e745c3cfb68474cc7ddf01fb8fd9b539e68a4d0169266d683142e3ca: Status 404 returned error can't find the container with id 613f00a6e745c3cfb68474cc7ddf01fb8fd9b539e68a4d0169266d683142e3ca
May 28 23:03:59.896108 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:03:59.896065    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-fsdnz" event={"ID":"9c347fe6-bfc7-4d0a-8974-92bb5edc26f9","Type":"ContainerStarted","Data":"613f00a6e745c3cfb68474cc7ddf01fb8fd9b539e68a4d0169266d683142e3ca"}
May 28 23:04:01.905876 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:01.905823    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-fsdnz" event={"ID":"9c347fe6-bfc7-4d0a-8974-92bb5edc26f9","Type":"ContainerStarted","Data":"d96e5b91aa65121fb40bb729e1b6659daf0cb0f1555f783719d22f63a50804b1"}
May 28 23:04:01.928778 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:01.928722    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-fsdnz" podStartSLOduration=2.365822987 podStartE2EDuration="3.928703373s" podCreationTimestamp="2026-05-28 23:03:58 +0000 UTC" firstStartedPulling="2026-05-28 23:03:59.404037989 +0000 UTC m=+112.597469220" lastFinishedPulling="2026-05-28 23:04:00.966918376 +0000 UTC m=+114.160349606" observedRunningTime="2026-05-28 23:04:01.928145402 +0000 UTC m=+115.121576653" watchObservedRunningTime="2026-05-28 23:04:01.928703373 +0000 UTC m=+115.122134626"
May 28 23:04:02.363977 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:02.363583    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x4gsd\" (UID: \"c93652d2-0a39-40de-ae3c-e31fa167e356\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"
May 28 23:04:02.363977 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:04:02.363756    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 28 23:04:02.363977 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:04:02.363818    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls podName:c93652d2-0a39-40de-ae3c-e31fa167e356 nodeName:}" failed. No retries permitted until 2026-05-28 23:04:18.363799901 +0000 UTC m=+131.557231150 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls") pod "cluster-samples-operator-697587599-x4gsd" (UID: "c93652d2-0a39-40de-ae3c-e31fa167e356") : secret "samples-operator-tls" not found
May 28 23:04:09.487742 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:09.487707    2577 scope.go:117] "RemoveContainer" containerID="dbc52b0aec12f732c2d704a635db2a8ada450e80cb95c52c9794267f19887df5"
May 28 23:04:09.924638 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:09.924608    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:04:09.925064 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:09.925043    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/1.log"
May 28 23:04:09.925147 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:09.925087    2577 generic.go:358] "Generic (PLEG): container finished" podID="edcd544f-4680-4190-a9b0-453053515b7a" containerID="88cf7b07a5940929885554c4fc98f10fda4df8f54efe2f2d2383853f5155fbe6" exitCode=255
May 28 23:04:09.925147 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:09.925131    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-nj29z" event={"ID":"edcd544f-4680-4190-a9b0-453053515b7a","Type":"ContainerDied","Data":"88cf7b07a5940929885554c4fc98f10fda4df8f54efe2f2d2383853f5155fbe6"}
May 28 23:04:09.925215 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:09.925171    2577 scope.go:117] "RemoveContainer" containerID="dbc52b0aec12f732c2d704a635db2a8ada450e80cb95c52c9794267f19887df5"
May 28 23:04:09.925502 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:09.925482    2577 scope.go:117] "RemoveContainer" containerID="88cf7b07a5940929885554c4fc98f10fda4df8f54efe2f2d2383853f5155fbe6"
May 28 23:04:09.925706 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:04:09.925680    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-nj29z_openshift-console-operator(edcd544f-4680-4190-a9b0-453053515b7a)\"" pod="openshift-console-operator/console-operator-76dff48d59-nj29z" podUID="edcd544f-4680-4190-a9b0-453053515b7a"
May 28 23:04:10.930158 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:10.930132    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:04:14.158186 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:14.158149    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xk7gh\" (UID: \"9d86f162-8349-42e1-abd1-aef9e7c328a4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:04:14.160820 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:14.160793    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d86f162-8349-42e1-abd1-aef9e7c328a4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xk7gh\" (UID: \"9d86f162-8349-42e1-abd1-aef9e7c328a4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:04:14.462614 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:14.462533    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6xtf6\""
May 28 23:04:14.470639 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:14.470614    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"
May 28 23:04:14.593888 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:14.593838    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh"]
May 28 23:04:14.596846 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:04:14.596816    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d86f162_8349_42e1_abd1_aef9e7c328a4.slice/crio-05e0cc33e17070f75985921af698e23a8a77c8fdde30d95714acf504802edeac WatchSource:0}: Error finding container 05e0cc33e17070f75985921af698e23a8a77c8fdde30d95714acf504802edeac: Status 404 returned error can't find the container with id 05e0cc33e17070f75985921af698e23a8a77c8fdde30d95714acf504802edeac
May 28 23:04:14.941125 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:14.941083    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh" event={"ID":"9d86f162-8349-42e1-abd1-aef9e7c328a4","Type":"ContainerStarted","Data":"05e0cc33e17070f75985921af698e23a8a77c8fdde30d95714acf504802edeac"}
May 28 23:04:16.174045 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:16.174005    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs\") pod \"network-metrics-daemon-mwvzm\" (UID: \"0a871ef7-90af-4a70-96ae-7df74d7ca072\") " pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:04:16.176744 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:16.176709    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a871ef7-90af-4a70-96ae-7df74d7ca072-metrics-certs\") pod \"network-metrics-daemon-mwvzm\" (UID: \"0a871ef7-90af-4a70-96ae-7df74d7ca072\") " pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:04:16.410588 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:16.410550    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-65lhq\""
May 28 23:04:16.417785 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:16.417758    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mwvzm"
May 28 23:04:16.656758 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:16.656728    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mwvzm"]
May 28 23:04:16.660554 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:04:16.660521    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a871ef7_90af_4a70_96ae_7df74d7ca072.slice/crio-ae54bab6ba6a3f5c6cfb89d0d1ac28da6a16f05f9f01c0763d99a33de24e4473 WatchSource:0}: Error finding container ae54bab6ba6a3f5c6cfb89d0d1ac28da6a16f05f9f01c0763d99a33de24e4473: Status 404 returned error can't find the container with id ae54bab6ba6a3f5c6cfb89d0d1ac28da6a16f05f9f01c0763d99a33de24e4473
May 28 23:04:16.946893 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:16.946784    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwvzm" event={"ID":"0a871ef7-90af-4a70-96ae-7df74d7ca072","Type":"ContainerStarted","Data":"ae54bab6ba6a3f5c6cfb89d0d1ac28da6a16f05f9f01c0763d99a33de24e4473"}
May 28 23:04:16.948059 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:16.948038    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh" event={"ID":"9d86f162-8349-42e1-abd1-aef9e7c328a4","Type":"ContainerStarted","Data":"b786dc7b8788d5a1cfbba5bcd646f7a2ea451d7b2b95f90e8b9e1ad4df19006a"}
May 28 23:04:16.965942 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:16.965905    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xk7gh" podStartSLOduration=32.980431837 podStartE2EDuration="34.965892941s" podCreationTimestamp="2026-05-28 23:03:42 +0000 UTC" firstStartedPulling="2026-05-28 23:04:14.598722674 +0000 UTC m=+127.792153911" lastFinishedPulling="2026-05-28 23:04:16.584183783 +0000 UTC m=+129.777615015" observedRunningTime="2026-05-28 23:04:16.965329259 +0000 UTC m=+130.158760514" watchObservedRunningTime="2026-05-28 23:04:16.965892941 +0000 UTC m=+130.159324192"
May 28 23:04:17.952895 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:17.952839    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwvzm" event={"ID":"0a871ef7-90af-4a70-96ae-7df74d7ca072","Type":"ContainerStarted","Data":"5cdae2bf251c5e1154fed422e24b0a66c75a5b4cae54f063f33572b35f13dc15"}
May 28 23:04:17.952895 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:17.952887    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mwvzm" event={"ID":"0a871ef7-90af-4a70-96ae-7df74d7ca072","Type":"ContainerStarted","Data":"2e88a8e965e0562f6bbcbdece4ec31306d449d10ee226f35cbabf3626e361b60"}
May 28 23:04:17.969114 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:17.969060    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mwvzm" podStartSLOduration=129.846894204 podStartE2EDuration="2m10.969040959s" podCreationTimestamp="2026-05-28 23:02:07 +0000 UTC" firstStartedPulling="2026-05-28 23:04:16.662556811 +0000 UTC m=+129.855988041" lastFinishedPulling="2026-05-28 23:04:17.784703566 +0000 UTC m=+130.978134796" observedRunningTime="2026-05-28 23:04:17.967447152 +0000 UTC m=+131.160878406" watchObservedRunningTime="2026-05-28 23:04:17.969040959 +0000 UTC m=+131.162472213"
May 28 23:04:18.391437 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:18.391380    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x4gsd\" (UID: \"c93652d2-0a39-40de-ae3c-e31fa167e356\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"
May 28 23:04:18.394036 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:18.394007    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c93652d2-0a39-40de-ae3c-e31fa167e356-samples-operator-tls\") pod \"cluster-samples-operator-697587599-x4gsd\" (UID: \"c93652d2-0a39-40de-ae3c-e31fa167e356\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"
May 28 23:04:18.649288 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:18.649208    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hz76d\""
May 28 23:04:18.657364 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:18.657344    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"
May 28 23:04:18.734619 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:18.734107    2577 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:04:18.734619 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:18.734151    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:04:18.734619 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:18.734517    2577 scope.go:117] "RemoveContainer" containerID="88cf7b07a5940929885554c4fc98f10fda4df8f54efe2f2d2383853f5155fbe6"
May 28 23:04:18.734900 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:04:18.734739    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-nj29z_openshift-console-operator(edcd544f-4680-4190-a9b0-453053515b7a)\"" pod="openshift-console-operator/console-operator-76dff48d59-nj29z" podUID="edcd544f-4680-4190-a9b0-453053515b7a"
May 28 23:04:18.796291 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:18.796258    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd"]
May 28 23:04:18.956281 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:18.956191    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd" event={"ID":"c93652d2-0a39-40de-ae3c-e31fa167e356","Type":"ContainerStarted","Data":"8415fa26994c9c66ce8216f0fb3ebd5737a0f6e67aeb50ab52793af0dda0e76e"}
May 28 23:04:20.963917 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:20.963886    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd" event={"ID":"c93652d2-0a39-40de-ae3c-e31fa167e356","Type":"ContainerStarted","Data":"e48368e085c06a66396b8310998af47aaf5f911c10185d30ddad510c6c6f48d6"}
May 28 23:04:20.964359 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:20.963925    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd" event={"ID":"c93652d2-0a39-40de-ae3c-e31fa167e356","Type":"ContainerStarted","Data":"4c5aeec021cb0d579f56606f1e855a791cc0cab4a62afbb57852b68b6406f932"}
May 28 23:04:20.981278 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:20.981226    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-x4gsd" podStartSLOduration=33.011280341 podStartE2EDuration="34.981212054s" podCreationTimestamp="2026-05-28 23:03:46 +0000 UTC" firstStartedPulling="2026-05-28 23:04:18.839585063 +0000 UTC m=+132.033016292" lastFinishedPulling="2026-05-28 23:04:20.809516767 +0000 UTC m=+134.002948005" observedRunningTime="2026-05-28 23:04:20.980430308 +0000 UTC m=+134.173861564" watchObservedRunningTime="2026-05-28 23:04:20.981212054 +0000 UTC m=+134.174643306"
May 28 23:04:21.647980 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.647949    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-d8mbh"]
May 28 23:04:21.650995 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.650972    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-d8mbh"
May 28 23:04:21.654463 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.654440    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
May 28 23:04:21.654463 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.654452    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
May 28 23:04:21.654605 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.654552    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9qb5m\""
May 28 23:04:21.659583 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.659559    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-d8mbh"]
May 28 23:04:21.778558 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.778526    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq"]
May 28 23:04:21.781594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.781573    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq"
May 28 23:04:21.784804 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.784785    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-q4hbd\""
May 28 23:04:21.786031 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.786016    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 28 23:04:21.810553 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.810525    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lx6p4"]
May 28 23:04:21.813602 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.813583    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq"]
May 28 23:04:21.813726 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.813711    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:21.816029 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.816009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b710a64b-729e-41cc-ab6a-fef55d47205d-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-d8mbh\" (UID: \"b710a64b-729e-41cc-ab6a-fef55d47205d\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-d8mbh"
May 28 23:04:21.816128 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.816061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b710a64b-729e-41cc-ab6a-fef55d47205d-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-d8mbh\" (UID: \"b710a64b-729e-41cc-ab6a-fef55d47205d\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-d8mbh"
May 28 23:04:21.816971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.816949    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 28 23:04:21.817045 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.817009    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 28 23:04:21.817588 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.817573    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 28 23:04:21.818186 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.818168    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 28 23:04:21.818486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.818472    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qd2hv\""
May 28 23:04:21.857973 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.857943    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lx6p4"]
May 28 23:04:21.917382 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.917312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2841d669-9492-43c9-a8e4-a19d6c7d6b00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx6p4\" (UID: \"2841d669-9492-43c9-a8e4-a19d6c7d6b00\") " pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:21.917382 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.917349    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b710a64b-729e-41cc-ab6a-fef55d47205d-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-d8mbh\" (UID: \"b710a64b-729e-41cc-ab6a-fef55d47205d\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-d8mbh"
May 28 23:04:21.917533 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.917394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b54001fb-0ec0-4dc2-8e17-8138b6ae0484-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq\" (UID: \"b54001fb-0ec0-4dc2-8e17-8138b6ae0484\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq"
May 28 23:04:21.917533 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.917491    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b710a64b-729e-41cc-ab6a-fef55d47205d-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-d8mbh\" (UID: \"b710a64b-729e-41cc-ab6a-fef55d47205d\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-d8mbh"
May 28 23:04:21.917533 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.917514    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2841d669-9492-43c9-a8e4-a19d6c7d6b00-data-volume\") pod \"insights-runtime-extractor-lx6p4\" (UID: \"2841d669-9492-43c9-a8e4-a19d6c7d6b00\") " pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:21.917653 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.917535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2841d669-9492-43c9-a8e4-a19d6c7d6b00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx6p4\" (UID: \"2841d669-9492-43c9-a8e4-a19d6c7d6b00\") " pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:21.917653 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.917568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6wgm\" (UniqueName: \"kubernetes.io/projected/2841d669-9492-43c9-a8e4-a19d6c7d6b00-kube-api-access-c6wgm\") pod \"insights-runtime-extractor-lx6p4\" (UID: \"2841d669-9492-43c9-a8e4-a19d6c7d6b00\") " pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:21.917653 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.917589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2841d669-9492-43c9-a8e4-a19d6c7d6b00-crio-socket\") pod \"insights-runtime-extractor-lx6p4\" (UID: \"2841d669-9492-43c9-a8e4-a19d6c7d6b00\") " pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:21.918157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.918138    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b710a64b-729e-41cc-ab6a-fef55d47205d-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-d8mbh\" (UID: \"b710a64b-729e-41cc-ab6a-fef55d47205d\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-d8mbh"
May 28 23:04:21.919838 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.919815    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b710a64b-729e-41cc-ab6a-fef55d47205d-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-d8mbh\" (UID: \"b710a64b-729e-41cc-ab6a-fef55d47205d\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-d8mbh"
May 28 23:04:21.960594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:21.960566    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-d8mbh"
May 28 23:04:22.018767 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.018541    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b54001fb-0ec0-4dc2-8e17-8138b6ae0484-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq\" (UID: \"b54001fb-0ec0-4dc2-8e17-8138b6ae0484\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq"
May 28 23:04:22.018767 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.018603    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2841d669-9492-43c9-a8e4-a19d6c7d6b00-data-volume\") pod \"insights-runtime-extractor-lx6p4\" (UID: \"2841d669-9492-43c9-a8e4-a19d6c7d6b00\") " pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:22.018767 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.018648    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2841d669-9492-43c9-a8e4-a19d6c7d6b00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx6p4\" (UID: \"2841d669-9492-43c9-a8e4-a19d6c7d6b00\") " pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:22.018767 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.018711    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c6wgm\" (UniqueName: \"kubernetes.io/projected/2841d669-9492-43c9-a8e4-a19d6c7d6b00-kube-api-access-c6wgm\") pod \"insights-runtime-extractor-lx6p4\" (UID: \"2841d669-9492-43c9-a8e4-a19d6c7d6b00\") " pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:22.018767 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.018738    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2841d669-9492-43c9-a8e4-a19d6c7d6b00-crio-socket\") pod \"insights-runtime-extractor-lx6p4\" (UID: \"2841d669-9492-43c9-a8e4-a19d6c7d6b00\") " pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:22.018767 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.018767    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2841d669-9492-43c9-a8e4-a19d6c7d6b00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx6p4\" (UID: \"2841d669-9492-43c9-a8e4-a19d6c7d6b00\") " pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:22.019498 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.018982    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2841d669-9492-43c9-a8e4-a19d6c7d6b00-data-volume\") pod \"insights-runtime-extractor-lx6p4\" (UID: \"2841d669-9492-43c9-a8e4-a19d6c7d6b00\") " pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:22.019498 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.019073    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2841d669-9492-43c9-a8e4-a19d6c7d6b00-crio-socket\") pod \"insights-runtime-extractor-lx6p4\" (UID: \"2841d669-9492-43c9-a8e4-a19d6c7d6b00\") " pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:22.019600 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.019529    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2841d669-9492-43c9-a8e4-a19d6c7d6b00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lx6p4\" (UID: \"2841d669-9492-43c9-a8e4-a19d6c7d6b00\") " pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:22.021766 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.021714    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2841d669-9492-43c9-a8e4-a19d6c7d6b00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lx6p4\" (UID: \"2841d669-9492-43c9-a8e4-a19d6c7d6b00\") " pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:22.022313 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.022275    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b54001fb-0ec0-4dc2-8e17-8138b6ae0484-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq\" (UID: \"b54001fb-0ec0-4dc2-8e17-8138b6ae0484\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq"
May 28 23:04:22.030476 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.030449    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6wgm\" (UniqueName: \"kubernetes.io/projected/2841d669-9492-43c9-a8e4-a19d6c7d6b00-kube-api-access-c6wgm\") pod \"insights-runtime-extractor-lx6p4\" (UID: \"2841d669-9492-43c9-a8e4-a19d6c7d6b00\") " pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:22.089820 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.089786    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq"
May 28 23:04:22.091518 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.091493    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-d8mbh"]
May 28 23:04:22.095512 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:04:22.095489    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb710a64b_729e_41cc_ab6a_fef55d47205d.slice/crio-e5f76d294ab23169fb0d729ea6c069cbddb965b041cd45b6a02538ae785fb20d WatchSource:0}: Error finding container e5f76d294ab23169fb0d729ea6c069cbddb965b041cd45b6a02538ae785fb20d: Status 404 returned error can't find the container with id e5f76d294ab23169fb0d729ea6c069cbddb965b041cd45b6a02538ae785fb20d
May 28 23:04:22.122777 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.122752    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lx6p4"
May 28 23:04:22.215939 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.215887    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq"]
May 28 23:04:22.219918 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:04:22.219885    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54001fb_0ec0_4dc2_8e17_8138b6ae0484.slice/crio-c798980243029308266bc06810fd30648906bdd4a3335e6582dd40ed6c18fb2f WatchSource:0}: Error finding container c798980243029308266bc06810fd30648906bdd4a3335e6582dd40ed6c18fb2f: Status 404 returned error can't find the container with id c798980243029308266bc06810fd30648906bdd4a3335e6582dd40ed6c18fb2f
May 28 23:04:22.269504 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.269473    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lx6p4"]
May 28 23:04:22.273047 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:04:22.273021    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2841d669_9492_43c9_a8e4_a19d6c7d6b00.slice/crio-2fe67bd8029793457920e760be62d9ae74c05a8bf468479cb8c57ad893794d1a WatchSource:0}: Error finding container 2fe67bd8029793457920e760be62d9ae74c05a8bf468479cb8c57ad893794d1a: Status 404 returned error can't find the container with id 2fe67bd8029793457920e760be62d9ae74c05a8bf468479cb8c57ad893794d1a
May 28 23:04:22.970941 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.970900    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-d8mbh" event={"ID":"b710a64b-729e-41cc-ab6a-fef55d47205d","Type":"ContainerStarted","Data":"e5f76d294ab23169fb0d729ea6c069cbddb965b041cd45b6a02538ae785fb20d"}
May 28 23:04:22.972394 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.972345    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lx6p4" event={"ID":"2841d669-9492-43c9-a8e4-a19d6c7d6b00","Type":"ContainerStarted","Data":"34297ec8b27227c5e10e4defbd8069b2bcfac0ec436311b3c4becafae56321c9"}
May 28 23:04:22.972561 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.972535    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lx6p4" event={"ID":"2841d669-9492-43c9-a8e4-a19d6c7d6b00","Type":"ContainerStarted","Data":"2fe67bd8029793457920e760be62d9ae74c05a8bf468479cb8c57ad893794d1a"}
May 28 23:04:22.973844 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:22.973807    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq" event={"ID":"b54001fb-0ec0-4dc2-8e17-8138b6ae0484","Type":"ContainerStarted","Data":"c798980243029308266bc06810fd30648906bdd4a3335e6582dd40ed6c18fb2f"}
May 28 23:04:23.977847 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:23.977763    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lx6p4" event={"ID":"2841d669-9492-43c9-a8e4-a19d6c7d6b00","Type":"ContainerStarted","Data":"1fc53f37674f7b406320c80748163a9e78be1811f32ef86d70980bb7835e27ca"}
May 28 23:04:23.979145 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:23.979121    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq" event={"ID":"b54001fb-0ec0-4dc2-8e17-8138b6ae0484","Type":"ContainerStarted","Data":"435cf30b68730566c923d6d0edc71d9b026533a1b2dd6e4fd31ffda66395f638"}
May 28 23:04:23.979349 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:23.979317    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq"
May 28 23:04:23.980540 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:23.980514    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-d8mbh" event={"ID":"b710a64b-729e-41cc-ab6a-fef55d47205d","Type":"ContainerStarted","Data":"2241eb54ef1ebf8452ad628771c62bf78b778e128ffb02552206617f6e827beb"}
May 28 23:04:23.984124 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:23.984103    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq"
May 28 23:04:23.994504 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:23.994458    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq" podStartSLOduration=1.60053108 podStartE2EDuration="2.994443095s" podCreationTimestamp="2026-05-28 23:04:21 +0000 UTC" firstStartedPulling="2026-05-28 23:04:22.22175055 +0000 UTC m=+135.415181780" lastFinishedPulling="2026-05-28 23:04:23.615662562 +0000 UTC m=+136.809093795" observedRunningTime="2026-05-28 23:04:23.993900318 +0000 UTC m=+137.187331566" watchObservedRunningTime="2026-05-28 23:04:23.994443095 +0000 UTC m=+137.187874388"
May 28 23:04:24.024952 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:24.024898    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-d8mbh" podStartSLOduration=1.508428308 podStartE2EDuration="3.024881609s" podCreationTimestamp="2026-05-28 23:04:21 +0000 UTC" firstStartedPulling="2026-05-28 23:04:22.097272308 +0000 UTC m=+135.290703538" lastFinishedPulling="2026-05-28 23:04:23.613725609 +0000 UTC m=+136.807156839" observedRunningTime="2026-05-28 23:04:24.02321348 +0000 UTC m=+137.216644733" watchObservedRunningTime="2026-05-28 23:04:24.024881609 +0000 UTC m=+137.218312853"
May 28 23:04:25.987450 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:25.987407    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lx6p4" event={"ID":"2841d669-9492-43c9-a8e4-a19d6c7d6b00","Type":"ContainerStarted","Data":"e3579b69450301aa7a1104b7f62dff01f65f266fef9bca338a4311f529662b41"}
May 28 23:04:26.010463 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:26.010414    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lx6p4" podStartSLOduration=2.402281601 podStartE2EDuration="5.01040078s" podCreationTimestamp="2026-05-28 23:04:21 +0000 UTC" firstStartedPulling="2026-05-28 23:04:22.331644885 +0000 UTC m=+135.525076115" lastFinishedPulling="2026-05-28 23:04:24.93976405 +0000 UTC m=+138.133195294" observedRunningTime="2026-05-28 23:04:26.010078154 +0000 UTC m=+139.203509407" watchObservedRunningTime="2026-05-28 23:04:26.01040078 +0000 UTC m=+139.203832032"
May 28 23:04:29.473091 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.473057    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"]
May 28 23:04:29.479952 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.479922    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"
May 28 23:04:29.483011 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.482975    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 28 23:04:29.483148 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.483020    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8zslv\""
May 28 23:04:29.483269 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.483250    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 28 23:04:29.483467 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.483347    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 28 23:04:29.483569 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.483543    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"]
May 28 23:04:29.488815 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.488796    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.494235 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.494214    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"]
May 28 23:04:29.498748 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.498722    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 28 23:04:29.499386 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.499358    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 28 23:04:29.505682 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.501013    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-t9ctk\""
May 28 23:04:29.507250 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.507234    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 28 23:04:29.511074 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.511008    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"]
May 28 23:04:29.514945 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.514922    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ns559"]
May 28 23:04:29.519102 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.519082    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.521492 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.521468    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 28 23:04:29.521619 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.521520    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kwtl7\""
May 28 23:04:29.521934 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.521904    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 28 23:04:29.522111 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.522095    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 28 23:04:29.574218 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.574185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsfb7\" (UniqueName: \"kubernetes.io/projected/c848eb78-984e-41f9-becd-7cdb443b1f51-kube-api-access-fsfb7\") pod \"openshift-state-metrics-9c959559f-w76gr\" (UID: \"c848eb78-984e-41f9-becd-7cdb443b1f51\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"
May 28 23:04:29.574396 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.574236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c848eb78-984e-41f9-becd-7cdb443b1f51-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-w76gr\" (UID: \"c848eb78-984e-41f9-becd-7cdb443b1f51\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"
May 28 23:04:29.574396 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.574331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c848eb78-984e-41f9-becd-7cdb443b1f51-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-w76gr\" (UID: \"c848eb78-984e-41f9-becd-7cdb443b1f51\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"
May 28 23:04:29.574396 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.574385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c848eb78-984e-41f9-becd-7cdb443b1f51-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-w76gr\" (UID: \"c848eb78-984e-41f9-becd-7cdb443b1f51\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"
May 28 23:04:29.674725 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.674691    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c848eb78-984e-41f9-becd-7cdb443b1f51-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-w76gr\" (UID: \"c848eb78-984e-41f9-becd-7cdb443b1f51\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"
May 28 23:04:29.674725 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.674736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-root\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.674989 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.674758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e3666ea-79b7-428b-bb95-96ff6ba5febc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.674989 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.674791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.674989 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.674851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bb7f\" (UniqueName: \"kubernetes.io/projected/4e3666ea-79b7-428b-bb95-96ff6ba5febc-kube-api-access-4bb7f\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.674989 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.674899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.674989 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.674922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4e3666ea-79b7-428b-bb95-96ff6ba5febc-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.674989 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.674938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e3666ea-79b7-428b-bb95-96ff6ba5febc-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.675294 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.675037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4e3666ea-79b7-428b-bb95-96ff6ba5febc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.675294 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.675084    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c848eb78-984e-41f9-becd-7cdb443b1f51-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-w76gr\" (UID: \"c848eb78-984e-41f9-becd-7cdb443b1f51\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"
May 28 23:04:29.675294 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.675114    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e3666ea-79b7-428b-bb95-96ff6ba5febc-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.675294 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.675148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-node-exporter-textfile\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.675294 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:04:29.675165    2577 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
May 28 23:04:29.675294 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.675186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-sys\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.675294 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:04:29.675211    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c848eb78-984e-41f9-becd-7cdb443b1f51-openshift-state-metrics-tls podName:c848eb78-984e-41f9-becd-7cdb443b1f51 nodeName:}" failed. No retries permitted until 2026-05-28 23:04:30.175196747 +0000 UTC m=+143.368627977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c848eb78-984e-41f9-becd-7cdb443b1f51-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-w76gr" (UID: "c848eb78-984e-41f9-becd-7cdb443b1f51") : secret "openshift-state-metrics-tls" not found
May 28 23:04:29.675294 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.675255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-node-exporter-wtmp\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.675294 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.675301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-node-exporter-tls\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.675673 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.675327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxnpn\" (UniqueName: \"kubernetes.io/projected/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-kube-api-access-jxnpn\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.675673 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.675381    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c848eb78-984e-41f9-becd-7cdb443b1f51-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-w76gr\" (UID: \"c848eb78-984e-41f9-becd-7cdb443b1f51\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"
May 28 23:04:29.675673 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.675427    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fsfb7\" (UniqueName: \"kubernetes.io/projected/c848eb78-984e-41f9-becd-7cdb443b1f51-kube-api-access-fsfb7\") pod \"openshift-state-metrics-9c959559f-w76gr\" (UID: \"c848eb78-984e-41f9-becd-7cdb443b1f51\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"
May 28 23:04:29.675673 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.675458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-metrics-client-ca\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.675673 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.675471    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c848eb78-984e-41f9-becd-7cdb443b1f51-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-w76gr\" (UID: \"c848eb78-984e-41f9-becd-7cdb443b1f51\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"
May 28 23:04:29.677994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.677965    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c848eb78-984e-41f9-becd-7cdb443b1f51-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-w76gr\" (UID: \"c848eb78-984e-41f9-becd-7cdb443b1f51\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"
May 28 23:04:29.684196 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.684163    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsfb7\" (UniqueName: \"kubernetes.io/projected/c848eb78-984e-41f9-becd-7cdb443b1f51-kube-api-access-fsfb7\") pod \"openshift-state-metrics-9c959559f-w76gr\" (UID: \"c848eb78-984e-41f9-becd-7cdb443b1f51\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"
May 28 23:04:29.776444 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.776409    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-root\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.776622 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.776453    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e3666ea-79b7-428b-bb95-96ff6ba5febc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.776622 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.776516    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-root\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.776622 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.776571    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.776622 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.776603    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4bb7f\" (UniqueName: \"kubernetes.io/projected/4e3666ea-79b7-428b-bb95-96ff6ba5febc-kube-api-access-4bb7f\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.776622 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.776620    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.776900 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.776645    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4e3666ea-79b7-428b-bb95-96ff6ba5febc-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.776900 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.776672    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e3666ea-79b7-428b-bb95-96ff6ba5febc-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.777024 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.776917    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4e3666ea-79b7-428b-bb95-96ff6ba5febc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.777024 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.776978    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e3666ea-79b7-428b-bb95-96ff6ba5febc-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.777024 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.777015    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-node-exporter-textfile\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.777167 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.777053    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-sys\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.777167 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.777066    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4e3666ea-79b7-428b-bb95-96ff6ba5febc-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.777167 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.777082    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-node-exporter-wtmp\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.777470 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.777220    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-node-exporter-wtmp\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.777470 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.777268    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-sys\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.777470 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.777273    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-node-exporter-textfile\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.777470 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.777327    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-node-exporter-tls\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.777470 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.777349    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jxnpn\" (UniqueName: \"kubernetes.io/projected/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-kube-api-access-jxnpn\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.777470 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.777379    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-metrics-client-ca\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.777764 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.777479    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.777764 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.777629    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4e3666ea-79b7-428b-bb95-96ff6ba5febc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.777897 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.777787    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-metrics-client-ca\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.777965 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.777911    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e3666ea-79b7-428b-bb95-96ff6ba5febc-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.779236 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.779214    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e3666ea-79b7-428b-bb95-96ff6ba5febc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.779678 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.779652    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e3666ea-79b7-428b-bb95-96ff6ba5febc-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.779881 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.779849    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-node-exporter-tls\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.780003 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.779985    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.785023 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.784996    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bb7f\" (UniqueName: \"kubernetes.io/projected/4e3666ea-79b7-428b-bb95-96ff6ba5febc-kube-api-access-4bb7f\") pod \"kube-state-metrics-54fd6bc5c4-bmlpg\" (UID: \"4e3666ea-79b7-428b-bb95-96ff6ba5febc\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.785417 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.785398    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxnpn\" (UniqueName: \"kubernetes.io/projected/4a8c8899-8fd2-4dfa-a219-41a0da90e36b-kube-api-access-jxnpn\") pod \"node-exporter-ns559\" (UID: \"4a8c8899-8fd2-4dfa-a219-41a0da90e36b\") " pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.816135 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.816093    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"
May 28 23:04:29.829901 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.829861    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ns559"
May 28 23:04:29.839453 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:04:29.839413    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8c8899_8fd2_4dfa_a219_41a0da90e36b.slice/crio-8827359d27d991c8b1249db8423feaa444891d2b0625cc3ebca2614dd38671f0 WatchSource:0}: Error finding container 8827359d27d991c8b1249db8423feaa444891d2b0625cc3ebca2614dd38671f0: Status 404 returned error can't find the container with id 8827359d27d991c8b1249db8423feaa444891d2b0625cc3ebca2614dd38671f0
May 28 23:04:29.941612 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.941572    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg"]
May 28 23:04:29.944132 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:04:29.944100    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3666ea_79b7_428b_bb95_96ff6ba5febc.slice/crio-3c5589e0088ad6720ac67be8c41ae2ad17a66243127d61fcd0699c2d3a77b178 WatchSource:0}: Error finding container 3c5589e0088ad6720ac67be8c41ae2ad17a66243127d61fcd0699c2d3a77b178: Status 404 returned error can't find the container with id 3c5589e0088ad6720ac67be8c41ae2ad17a66243127d61fcd0699c2d3a77b178
May 28 23:04:29.997547 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.997515    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ns559" event={"ID":"4a8c8899-8fd2-4dfa-a219-41a0da90e36b","Type":"ContainerStarted","Data":"8827359d27d991c8b1249db8423feaa444891d2b0625cc3ebca2614dd38671f0"}
May 28 23:04:29.998473 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:29.998449    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg" event={"ID":"4e3666ea-79b7-428b-bb95-96ff6ba5febc","Type":"ContainerStarted","Data":"3c5589e0088ad6720ac67be8c41ae2ad17a66243127d61fcd0699c2d3a77b178"}
May 28 23:04:30.181320 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.181222    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c848eb78-984e-41f9-becd-7cdb443b1f51-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-w76gr\" (UID: \"c848eb78-984e-41f9-becd-7cdb443b1f51\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"
May 28 23:04:30.183737 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.183705    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c848eb78-984e-41f9-becd-7cdb443b1f51-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-w76gr\" (UID: \"c848eb78-984e-41f9-becd-7cdb443b1f51\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"
May 28 23:04:30.400993 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.400958    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"
May 28 23:04:30.487550 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.487511    2577 scope.go:117] "RemoveContainer" containerID="88cf7b07a5940929885554c4fc98f10fda4df8f54efe2f2d2383853f5155fbe6"
May 28 23:04:30.573594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.573567    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-w76gr"]
May 28 23:04:30.577353 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:04:30.577320    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc848eb78_984e_41f9_becd_7cdb443b1f51.slice/crio-9521eb9914708c71a3acc7beed0db3901413cec5d4b7c4cc754677d9acb1fc91 WatchSource:0}: Error finding container 9521eb9914708c71a3acc7beed0db3901413cec5d4b7c4cc754677d9acb1fc91: Status 404 returned error can't find the container with id 9521eb9914708c71a3acc7beed0db3901413cec5d4b7c4cc754677d9acb1fc91
May 28 23:04:30.583088 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.581102    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 23:04:30.586038 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.585675    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.589016 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.588826    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 28 23:04:30.589917 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.589749    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 28 23:04:30.590786 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.590024    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 28 23:04:30.590786 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.590282    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 28 23:04:30.590786 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.590344    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 28 23:04:30.590786 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.590481    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 28 23:04:30.590786 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.590483    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 28 23:04:30.590786 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.590577    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 28 23:04:30.591442 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.590914    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 28 23:04:30.591442 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.590965    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zqxhv\""
May 28 23:04:30.599079 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.599040    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 23:04:30.686293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.685969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.686293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.686027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.686293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.686058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.686293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.686091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.686293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.686119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.686293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.686145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3a42853-d4b8-4c40-92ad-c991c665f9bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.686293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.686168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzfqv\" (UniqueName: \"kubernetes.io/projected/f3a42853-d4b8-4c40-92ad-c991c665f9bf-kube-api-access-lzfqv\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.686293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.686220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.686892 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.686322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3a42853-d4b8-4c40-92ad-c991c665f9bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.686892 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.686365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3a42853-d4b8-4c40-92ad-c991c665f9bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.686892 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.686400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3a42853-d4b8-4c40-92ad-c991c665f9bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.686892 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.686426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3a42853-d4b8-4c40-92ad-c991c665f9bf-config-out\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.686892 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.686460    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-web-config\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.788326 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.787636    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.788326 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.787691    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.788326 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.787721    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.788326 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.787747    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3a42853-d4b8-4c40-92ad-c991c665f9bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.788326 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.787772    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lzfqv\" (UniqueName: \"kubernetes.io/projected/f3a42853-d4b8-4c40-92ad-c991c665f9bf-kube-api-access-lzfqv\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.788326 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:04:30.787780    2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 28 23:04:30.788326 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.787797    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.788326 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:04:30.787857    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-main-tls podName:f3a42853-d4b8-4c40-92ad-c991c665f9bf nodeName:}" failed. No retries permitted until 2026-05-28 23:04:31.287833921 +0000 UTC m=+144.481265155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f3a42853-d4b8-4c40-92ad-c991c665f9bf") : secret "alertmanager-main-tls" not found
May 28 23:04:30.788326 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.787923    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3a42853-d4b8-4c40-92ad-c991c665f9bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.788326 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.787952    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3a42853-d4b8-4c40-92ad-c991c665f9bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.788326 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.787988    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3a42853-d4b8-4c40-92ad-c991c665f9bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.788326 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.788016    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3a42853-d4b8-4c40-92ad-c991c665f9bf-config-out\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.788326 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.788052    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-web-config\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.788326 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.788117    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.788326 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.788165    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.789544 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.788416    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3a42853-d4b8-4c40-92ad-c991c665f9bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.789544 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.789089    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3a42853-d4b8-4c40-92ad-c991c665f9bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.790284 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.790254    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3a42853-d4b8-4c40-92ad-c991c665f9bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.795357 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.795219    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.796648 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.796582    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.797178 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.797132    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3a42853-d4b8-4c40-92ad-c991c665f9bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.800326 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.800283    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.801686 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.801435    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3a42853-d4b8-4c40-92ad-c991c665f9bf-config-out\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.804124 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.804033    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-web-config\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.805116 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.804752    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.805646 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.805608    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:30.807282 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:30.807241    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzfqv\" (UniqueName: \"kubernetes.io/projected/f3a42853-d4b8-4c40-92ad-c991c665f9bf-kube-api-access-lzfqv\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:31.005671 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.004950    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:04:31.005671 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.005109    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-nj29z" event={"ID":"edcd544f-4680-4190-a9b0-453053515b7a","Type":"ContainerStarted","Data":"88af99a8be7b3685f2c4fdec91d7b77bc0f50eab63fb427a66536a5524c264aa"}
May 28 23:04:31.005908 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.005802    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:04:31.007968 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.007945    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr" event={"ID":"c848eb78-984e-41f9-becd-7cdb443b1f51","Type":"ContainerStarted","Data":"2c2d8b9cd44cc28f7a5df4a67d6ca127d6aeadc03a696a6f4bae031c83090c09"}
May 28 23:04:31.008079 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.007974    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr" event={"ID":"c848eb78-984e-41f9-becd-7cdb443b1f51","Type":"ContainerStarted","Data":"9521eb9914708c71a3acc7beed0db3901413cec5d4b7c4cc754677d9acb1fc91"}
May 28 23:04:31.023978 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.023596    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-nj29z" podStartSLOduration=40.86572234 podStartE2EDuration="43.023580839s" podCreationTimestamp="2026-05-28 23:03:48 +0000 UTC" firstStartedPulling="2026-05-28 23:03:48.856831242 +0000 UTC m=+102.050262473" lastFinishedPulling="2026-05-28 23:03:51.01468973 +0000 UTC m=+104.208120972" observedRunningTime="2026-05-28 23:04:31.022652949 +0000 UTC m=+144.216084198" watchObservedRunningTime="2026-05-28 23:04:31.023580839 +0000 UTC m=+144.217012092"
May 28 23:04:31.294933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.294840    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:31.295086 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:04:31.295001    2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 28 23:04:31.295086 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:04:31.295082    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-main-tls podName:f3a42853-d4b8-4c40-92ad-c991c665f9bf nodeName:}" failed. No retries permitted until 2026-05-28 23:04:32.295059795 +0000 UTC m=+145.488491042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f3a42853-d4b8-4c40-92ad-c991c665f9bf") : secret "alertmanager-main-tls" not found
May 28 23:04:31.323552 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.323472    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-nj29z"
May 28 23:04:31.497979 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.497942    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-6xh5k"]
May 28 23:04:31.501165 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.501139    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-6xh5k"
May 28 23:04:31.504388 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.503999    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-g98mr\""
May 28 23:04:31.504388 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.504218    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 28 23:04:31.504388 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.504222    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 28 23:04:31.512733 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.512709    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-6xh5k"]
May 28 23:04:31.597317 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.597285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kpdl\" (UniqueName: \"kubernetes.io/projected/94ca47b1-b8a4-47f6-b3ac-b4f806bf1d0e-kube-api-access-9kpdl\") pod \"downloads-57bd96b888-6xh5k\" (UID: \"94ca47b1-b8a4-47f6-b3ac-b4f806bf1d0e\") " pod="openshift-console/downloads-57bd96b888-6xh5k"
May 28 23:04:31.698099 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.698055    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9kpdl\" (UniqueName: \"kubernetes.io/projected/94ca47b1-b8a4-47f6-b3ac-b4f806bf1d0e-kube-api-access-9kpdl\") pod \"downloads-57bd96b888-6xh5k\" (UID: \"94ca47b1-b8a4-47f6-b3ac-b4f806bf1d0e\") " pod="openshift-console/downloads-57bd96b888-6xh5k"
May 28 23:04:31.708937 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.708905    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kpdl\" (UniqueName: \"kubernetes.io/projected/94ca47b1-b8a4-47f6-b3ac-b4f806bf1d0e-kube-api-access-9kpdl\") pod \"downloads-57bd96b888-6xh5k\" (UID: \"94ca47b1-b8a4-47f6-b3ac-b4f806bf1d0e\") " pod="openshift-console/downloads-57bd96b888-6xh5k"
May 28 23:04:31.814152 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.813614    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-6xh5k"
May 28 23:04:31.972404 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:31.972347    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-6xh5k"]
May 28 23:04:31.975485 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:04:31.975449    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ca47b1_b8a4_47f6_b3ac_b4f806bf1d0e.slice/crio-0af176afb0bb499be01c635d3d33986ab9ef699b3559d7b1fc508335af720a43 WatchSource:0}: Error finding container 0af176afb0bb499be01c635d3d33986ab9ef699b3559d7b1fc508335af720a43: Status 404 returned error can't find the container with id 0af176afb0bb499be01c635d3d33986ab9ef699b3559d7b1fc508335af720a43
May 28 23:04:32.012944 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:32.012909    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr" event={"ID":"c848eb78-984e-41f9-becd-7cdb443b1f51","Type":"ContainerStarted","Data":"d7e3df4cccfcf3b292ed9433cd0d7fcd42a9c39836a8578813ff06abebe3aa86"}
May 28 23:04:32.014170 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:32.014146    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-6xh5k" event={"ID":"94ca47b1-b8a4-47f6-b3ac-b4f806bf1d0e","Type":"ContainerStarted","Data":"0af176afb0bb499be01c635d3d33986ab9ef699b3559d7b1fc508335af720a43"}
May 28 23:04:32.015847 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:32.015822    2577 generic.go:358] "Generic (PLEG): container finished" podID="4a8c8899-8fd2-4dfa-a219-41a0da90e36b" containerID="0cf31987881b03293ea00c7d365048d51c377c069ecfb0754e7d9311ca395c8c" exitCode=0
May 28 23:04:32.015949 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:32.015907    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ns559" event={"ID":"4a8c8899-8fd2-4dfa-a219-41a0da90e36b","Type":"ContainerDied","Data":"0cf31987881b03293ea00c7d365048d51c377c069ecfb0754e7d9311ca395c8c"}
May 28 23:04:32.018059 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:32.018033    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg" event={"ID":"4e3666ea-79b7-428b-bb95-96ff6ba5febc","Type":"ContainerStarted","Data":"c20c23e246f2c72bc0ec9308ec5902ecc41c8a4be9d4dffafa71e6c21d695067"}
May 28 23:04:32.018159 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:32.018069    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg" event={"ID":"4e3666ea-79b7-428b-bb95-96ff6ba5febc","Type":"ContainerStarted","Data":"2cd675f94fea2e3024e321a29e4f6256df59ebe1bb5550c10d45c74f29fbb875"}
May 28 23:04:32.018159 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:32.018086    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg" event={"ID":"4e3666ea-79b7-428b-bb95-96ff6ba5febc","Type":"ContainerStarted","Data":"631333f5bea67cdf92b62b69229d4ee76a34e448fad5c558d981ba5adfc0ccd7"}
May 28 23:04:32.056128 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:32.056067    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-bmlpg" podStartSLOduration=1.441268235 podStartE2EDuration="3.056048547s" podCreationTimestamp="2026-05-28 23:04:29 +0000 UTC" firstStartedPulling="2026-05-28 23:04:29.945813667 +0000 UTC m=+143.139244897" lastFinishedPulling="2026-05-28 23:04:31.560593962 +0000 UTC m=+144.754025209" observedRunningTime="2026-05-28 23:04:32.055231435 +0000 UTC m=+145.248662720" watchObservedRunningTime="2026-05-28 23:04:32.056048547 +0000 UTC m=+145.249479800"
May 28 23:04:32.305054 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:32.305031    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:32.307494 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:32.307469    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:32.398830 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:32.398793    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 23:04:32.585901 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:32.585039    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 23:04:33.023558 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.023513    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ns559" event={"ID":"4a8c8899-8fd2-4dfa-a219-41a0da90e36b","Type":"ContainerStarted","Data":"f82d5a0b63d83b43ecfdf3577301436b974d53c6ffd8486194a4c066b2269d8b"}
May 28 23:04:33.023558 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.023561    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ns559" event={"ID":"4a8c8899-8fd2-4dfa-a219-41a0da90e36b","Type":"ContainerStarted","Data":"01e1ba5d3c9207d95520e7ea73ff2f4b54060da9a7a3f5a004a72570d63b8a23"}
May 28 23:04:33.024744 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.024715    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a42853-d4b8-4c40-92ad-c991c665f9bf","Type":"ContainerStarted","Data":"d7ea680aa5a7d6e36812afce699f3e53553c6d0e1a8d8f8547aef38a88737240"}
May 28 23:04:33.026810 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.026775    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr" event={"ID":"c848eb78-984e-41f9-becd-7cdb443b1f51","Type":"ContainerStarted","Data":"a6cd2096db327fa77a344b3bf7466e06f3618709dcbcac93cabb20502e5d2232"}
May 28 23:04:33.043254 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.043203    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ns559" podStartSLOduration=2.8719193990000003 podStartE2EDuration="4.043186747s" podCreationTimestamp="2026-05-28 23:04:29 +0000 UTC" firstStartedPulling="2026-05-28 23:04:29.841138243 +0000 UTC m=+143.034569473" lastFinishedPulling="2026-05-28 23:04:31.012405585 +0000 UTC m=+144.205836821" observedRunningTime="2026-05-28 23:04:33.04162342 +0000 UTC m=+146.235054671" watchObservedRunningTime="2026-05-28 23:04:33.043186747 +0000 UTC m=+146.236618043"
May 28 23:04:33.058675 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.058629    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-w76gr" podStartSLOduration=2.832771924 podStartE2EDuration="4.058613853s" podCreationTimestamp="2026-05-28 23:04:29 +0000 UTC" firstStartedPulling="2026-05-28 23:04:31.027592689 +0000 UTC m=+144.221023934" lastFinishedPulling="2026-05-28 23:04:32.253434633 +0000 UTC m=+145.446865863" observedRunningTime="2026-05-28 23:04:33.057391062 +0000 UTC m=+146.250822314" watchObservedRunningTime="2026-05-28 23:04:33.058613853 +0000 UTC m=+146.252045108"
May 28 23:04:33.891479 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.891447    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5bc944fdd7-djthh"]
May 28 23:04:33.894663 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.894639    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:33.897327 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.897304    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 28 23:04:33.897440 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.897306    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 28 23:04:33.898227 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.898204    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8uq0k6vhvj9ft\""
May 28 23:04:33.898322 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.898285    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 28 23:04:33.898384 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.898337    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 28 23:04:33.898438 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.898285    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bhcml\""
May 28 23:04:33.902590 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.902568    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5bc944fdd7-djthh"]
May 28 23:04:33.919626 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.919601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f5528f72-6719-4181-b465-d1f8d01e2431-audit-log\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:33.919765 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.919646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5528f72-6719-4181-b465-d1f8d01e2431-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:33.919765 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.919684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f5528f72-6719-4181-b465-d1f8d01e2431-secret-metrics-server-tls\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:33.919765 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.919724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f5528f72-6719-4181-b465-d1f8d01e2431-metrics-server-audit-profiles\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:33.919765 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.919754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f5528f72-6719-4181-b465-d1f8d01e2431-client-ca-bundle\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:33.919994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.919804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5gwz\" (UniqueName: \"kubernetes.io/projected/f5528f72-6719-4181-b465-d1f8d01e2431-kube-api-access-v5gwz\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:33.919994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:33.919845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f5528f72-6719-4181-b465-d1f8d01e2431-secret-metrics-server-client-certs\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.020375 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.020338    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f5528f72-6719-4181-b465-d1f8d01e2431-secret-metrics-server-client-certs\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.020577 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.020430    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f5528f72-6719-4181-b465-d1f8d01e2431-audit-log\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.020577 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.020467    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5528f72-6719-4181-b465-d1f8d01e2431-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.020577 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.020504    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f5528f72-6719-4181-b465-d1f8d01e2431-secret-metrics-server-tls\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.020577 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.020546    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f5528f72-6719-4181-b465-d1f8d01e2431-metrics-server-audit-profiles\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.020577 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.020573    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f5528f72-6719-4181-b465-d1f8d01e2431-client-ca-bundle\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.020829 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.020610    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gwz\" (UniqueName: \"kubernetes.io/projected/f5528f72-6719-4181-b465-d1f8d01e2431-kube-api-access-v5gwz\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.021525 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.021078    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f5528f72-6719-4181-b465-d1f8d01e2431-audit-log\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.021525 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.021479    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5528f72-6719-4181-b465-d1f8d01e2431-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.021686 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.021675    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f5528f72-6719-4181-b465-d1f8d01e2431-metrics-server-audit-profiles\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.023575 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.023545    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f5528f72-6719-4181-b465-d1f8d01e2431-secret-metrics-server-client-certs\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.023684 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.023645    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f5528f72-6719-4181-b465-d1f8d01e2431-secret-metrics-server-tls\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.023684 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.023678    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f5528f72-6719-4181-b465-d1f8d01e2431-client-ca-bundle\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.030117 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.030090    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gwz\" (UniqueName: \"kubernetes.io/projected/f5528f72-6719-4181-b465-d1f8d01e2431-kube-api-access-v5gwz\") pod \"metrics-server-5bc944fdd7-djthh\" (UID: \"f5528f72-6719-4181-b465-d1f8d01e2431\") " pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.031399 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.031370    2577 generic.go:358] "Generic (PLEG): container finished" podID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerID="82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199" exitCode=0
May 28 23:04:34.031519 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.031461    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a42853-d4b8-4c40-92ad-c991c665f9bf","Type":"ContainerDied","Data":"82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199"}
May 28 23:04:34.254954 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.254901    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:34.429546 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:34.429511    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5bc944fdd7-djthh"]
May 28 23:04:34.432352 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:04:34.432318    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5528f72_6719_4181_b465_d1f8d01e2431.slice/crio-72ab7c31fa72fb290088a8f8ac36537ae295112e18b58c99e9176bb1a81e8757 WatchSource:0}: Error finding container 72ab7c31fa72fb290088a8f8ac36537ae295112e18b58c99e9176bb1a81e8757: Status 404 returned error can't find the container with id 72ab7c31fa72fb290088a8f8ac36537ae295112e18b58c99e9176bb1a81e8757
May 28 23:04:35.036318 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:35.036259    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh" event={"ID":"f5528f72-6719-4181-b465-d1f8d01e2431","Type":"ContainerStarted","Data":"72ab7c31fa72fb290088a8f8ac36537ae295112e18b58c99e9176bb1a81e8757"}
May 28 23:04:36.045850 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:36.045808    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a42853-d4b8-4c40-92ad-c991c665f9bf","Type":"ContainerStarted","Data":"6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6"}
May 28 23:04:37.052231 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:37.052168    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh" event={"ID":"f5528f72-6719-4181-b465-d1f8d01e2431","Type":"ContainerStarted","Data":"c2d6f8ca7a993e5bebdae2b2d7757c051947a3f2bc0c4cb617c52e0e7018e289"}
May 28 23:04:37.055012 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:37.054986    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a42853-d4b8-4c40-92ad-c991c665f9bf","Type":"ContainerStarted","Data":"2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e"}
May 28 23:04:37.055012 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:37.055015    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a42853-d4b8-4c40-92ad-c991c665f9bf","Type":"ContainerStarted","Data":"6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f"}
May 28 23:04:37.055193 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:37.055027    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a42853-d4b8-4c40-92ad-c991c665f9bf","Type":"ContainerStarted","Data":"cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27"}
May 28 23:04:37.055193 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:37.055039    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a42853-d4b8-4c40-92ad-c991c665f9bf","Type":"ContainerStarted","Data":"2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e"}
May 28 23:04:37.069615 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:37.069560    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh" podStartSLOduration=2.265210164 podStartE2EDuration="4.069544024s" podCreationTimestamp="2026-05-28 23:04:33 +0000 UTC" firstStartedPulling="2026-05-28 23:04:34.434485577 +0000 UTC m=+147.627916807" lastFinishedPulling="2026-05-28 23:04:36.238819422 +0000 UTC m=+149.432250667" observedRunningTime="2026-05-28 23:04:37.069159424 +0000 UTC m=+150.262590672" watchObservedRunningTime="2026-05-28 23:04:37.069544024 +0000 UTC m=+150.262975277"
May 28 23:04:38.062225 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:38.062187    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a42853-d4b8-4c40-92ad-c991c665f9bf","Type":"ContainerStarted","Data":"d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f"}
May 28 23:04:38.089373 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:38.089320    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.568610784 podStartE2EDuration="8.089305563s" podCreationTimestamp="2026-05-28 23:04:30 +0000 UTC" firstStartedPulling="2026-05-28 23:04:32.595160362 +0000 UTC m=+145.788591605" lastFinishedPulling="2026-05-28 23:04:37.11585515 +0000 UTC m=+150.309286384" observedRunningTime="2026-05-28 23:04:38.08699303 +0000 UTC m=+151.280424283" watchObservedRunningTime="2026-05-28 23:04:38.089305563 +0000 UTC m=+151.282736815"
May 28 23:04:43.023422 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.023383    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5589799f6d-qr8d4"]
May 28 23:04:43.028050 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.028020    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.030582 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.030558    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 28 23:04:43.030721 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.030691    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 28 23:04:43.031498 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.031475    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pqd4t\""
May 28 23:04:43.031612 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.031560    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 28 23:04:43.031677 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.031627    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 28 23:04:43.031737 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.031690    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 28 23:04:43.034589 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.034565    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5589799f6d-qr8d4"]
May 28 23:04:43.106756 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.106720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq95d\" (UniqueName: \"kubernetes.io/projected/7d5a9573-d1e5-4894-a60c-2db990e96ed0-kube-api-access-jq95d\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.106939 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.106764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-oauth-config\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.106939 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.106843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-oauth-serving-cert\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.106939 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.106925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-serving-cert\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.107088 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.106971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-config\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.107142 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.107075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-service-ca\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.208452 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.208413    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jq95d\" (UniqueName: \"kubernetes.io/projected/7d5a9573-d1e5-4894-a60c-2db990e96ed0-kube-api-access-jq95d\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.208640 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.208464    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-oauth-config\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.208640 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.208553    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-oauth-serving-cert\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.208763 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.208646    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-serving-cert\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.208763 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.208703    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-config\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.208763 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.208753    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-service-ca\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.209339 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.209293    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-oauth-serving-cert\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.209546 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.209513    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-service-ca\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.209718 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.209697    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-config\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.211926 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.211902    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-oauth-config\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.211926 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.211920    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-serving-cert\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.217174 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.217152    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq95d\" (UniqueName: \"kubernetes.io/projected/7d5a9573-d1e5-4894-a60c-2db990e96ed0-kube-api-access-jq95d\") pod \"console-5589799f6d-qr8d4\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") " pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.341099 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:43.341007    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:04:43.705361 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:04:43.705272    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-n77fp" podUID="0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b"
May 28 23:04:43.718861 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:04:43.718825    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-47c8w" podUID="ba8f16cc-755b-4dc9-a1b5-3c33e7149c77"
May 28 23:04:44.081587 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:44.081554    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n77fp"
May 28 23:04:48.560042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:48.560006    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:04:48.560514 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:48.560102    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert\") pod \"ingress-canary-47c8w\" (UID: \"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77\") " pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:04:48.562834 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:48.562802    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b-metrics-tls\") pod \"dns-default-n77fp\" (UID: \"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b\") " pod="openshift-dns/dns-default-n77fp"
May 28 23:04:48.562985 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:48.562896    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba8f16cc-755b-4dc9-a1b5-3c33e7149c77-cert\") pod \"ingress-canary-47c8w\" (UID: \"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77\") " pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:04:48.585891 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:48.585848    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-94qps\""
May 28 23:04:48.593020 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:48.592998    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n77fp"
May 28 23:04:49.895113 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:49.894959    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5589799f6d-qr8d4"]
May 28 23:04:49.898481 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:04:49.898444    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5a9573_d1e5_4894_a60c_2db990e96ed0.slice/crio-d1dff1636da6dee6816f3f352d429245d41b5aaf41d7f574af190f31eb58e83f WatchSource:0}: Error finding container d1dff1636da6dee6816f3f352d429245d41b5aaf41d7f574af190f31eb58e83f: Status 404 returned error can't find the container with id d1dff1636da6dee6816f3f352d429245d41b5aaf41d7f574af190f31eb58e83f
May 28 23:04:49.911945 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:49.911855    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n77fp"]
May 28 23:04:49.917842 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:04:49.917815    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f654df9_6c79_4a7f_9d6d_6933e9bdfa5b.slice/crio-f7cf9ddec73ec3a1a2cb8c90393871c542527c27f475a9fbef3ee2bdac4bcc76 WatchSource:0}: Error finding container f7cf9ddec73ec3a1a2cb8c90393871c542527c27f475a9fbef3ee2bdac4bcc76: Status 404 returned error can't find the container with id f7cf9ddec73ec3a1a2cb8c90393871c542527c27f475a9fbef3ee2bdac4bcc76
May 28 23:04:50.101474 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:50.101390    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5589799f6d-qr8d4" event={"ID":"7d5a9573-d1e5-4894-a60c-2db990e96ed0","Type":"ContainerStarted","Data":"d1dff1636da6dee6816f3f352d429245d41b5aaf41d7f574af190f31eb58e83f"}
May 28 23:04:50.103132 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:50.103094    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-6xh5k" event={"ID":"94ca47b1-b8a4-47f6-b3ac-b4f806bf1d0e","Type":"ContainerStarted","Data":"be412a16618ae03aaf8963251861ced3028c7509a8e69a832e296735a0552f6e"}
May 28 23:04:50.103493 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:50.103458    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-6xh5k"
May 28 23:04:50.104394 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:50.104362    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n77fp" event={"ID":"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b","Type":"ContainerStarted","Data":"f7cf9ddec73ec3a1a2cb8c90393871c542527c27f475a9fbef3ee2bdac4bcc76"}
May 28 23:04:50.104785 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:50.104752    2577 patch_prober.go:28] interesting pod/downloads-57bd96b888-6xh5k container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.21:8080/\": dial tcp 10.133.0.21:8080: connect: connection refused" start-of-body=
May 28 23:04:50.104915 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:50.104820    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-57bd96b888-6xh5k" podUID="94ca47b1-b8a4-47f6-b3ac-b4f806bf1d0e" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.21:8080/\": dial tcp 10.133.0.21:8080: connect: connection refused"
May 28 23:04:50.121127 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:50.121072    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-6xh5k" podStartSLOduration=1.245871596 podStartE2EDuration="19.121055395s" podCreationTimestamp="2026-05-28 23:04:31 +0000 UTC" firstStartedPulling="2026-05-28 23:04:31.977632425 +0000 UTC m=+145.171063659" lastFinishedPulling="2026-05-28 23:04:49.852816228 +0000 UTC m=+163.046247458" observedRunningTime="2026-05-28 23:04:50.119488554 +0000 UTC m=+163.312919809" watchObservedRunningTime="2026-05-28 23:04:50.121055395 +0000 UTC m=+163.314486648"
May 28 23:04:51.123713 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:51.123656    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-6xh5k"
May 28 23:04:53.404890 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.404829    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-55985d6dd6-ph7sg"]
May 28 23:04:53.475560 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.475516    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55985d6dd6-ph7sg"]
May 28 23:04:53.475790 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.475659    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.484296 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.484264    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 28 23:04:53.607340 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.607307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-console-config\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.607465 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.607360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef7edb4-9565-433d-89d2-0abcbd687401-console-serving-cert\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.607546 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.607459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vrmh\" (UniqueName: \"kubernetes.io/projected/9ef7edb4-9565-433d-89d2-0abcbd687401-kube-api-access-2vrmh\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.607546 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.607494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-oauth-serving-cert\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.607546 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.607533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-service-ca\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.607703 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.607629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-trusted-ca-bundle\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.607703 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.607669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ef7edb4-9565-433d-89d2-0abcbd687401-console-oauth-config\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.708288 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.708254    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef7edb4-9565-433d-89d2-0abcbd687401-console-serving-cert\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.708410 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.708343    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2vrmh\" (UniqueName: \"kubernetes.io/projected/9ef7edb4-9565-433d-89d2-0abcbd687401-kube-api-access-2vrmh\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.708410 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.708375    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-oauth-serving-cert\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.708410 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.708407    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-service-ca\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.708595 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.708470    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-trusted-ca-bundle\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.708595 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.708503    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ef7edb4-9565-433d-89d2-0abcbd687401-console-oauth-config\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.708595 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.708563    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-console-config\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.711890 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.711845    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-console-config\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.713676 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.712402    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-service-ca\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.713676 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.712884    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-oauth-serving-cert\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.713676 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.713362    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef7edb4-9565-433d-89d2-0abcbd687401-console-serving-cert\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.722938 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.721950    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ef7edb4-9565-433d-89d2-0abcbd687401-console-oauth-config\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.724892 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.724838    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-trusted-ca-bundle\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.726975 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.726840    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vrmh\" (UniqueName: \"kubernetes.io/projected/9ef7edb4-9565-433d-89d2-0abcbd687401-kube-api-access-2vrmh\") pod \"console-55985d6dd6-ph7sg\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") " pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:53.790161 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:53.790121    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:04:54.114744 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:54.114573    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55985d6dd6-ph7sg"]
May 28 23:04:54.118901 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:04:54.118739    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef7edb4_9565_433d_89d2_0abcbd687401.slice/crio-dacdc3e722cd82d3b951ac0b96ea44aa9b9dc653520481662c6fa6c1ae464c40 WatchSource:0}: Error finding container dacdc3e722cd82d3b951ac0b96ea44aa9b9dc653520481662c6fa6c1ae464c40: Status 404 returned error can't find the container with id dacdc3e722cd82d3b951ac0b96ea44aa9b9dc653520481662c6fa6c1ae464c40
May 28 23:04:54.122076 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:54.122042    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n77fp" event={"ID":"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b","Type":"ContainerStarted","Data":"f426fc0139d325671f911d13d8c1913e823a693cf78480cca4f6e28d84389bc9"}
May 28 23:04:54.256355 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:54.255970    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:54.256355 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:54.256298    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:04:55.127045 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:55.126920    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55985d6dd6-ph7sg" event={"ID":"9ef7edb4-9565-433d-89d2-0abcbd687401","Type":"ContainerStarted","Data":"4e3d394c8b09eb61d7d8a98fceeab6cea4f7c517fd0385860b65048d1844004c"}
May 28 23:04:55.127045 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:55.126972    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55985d6dd6-ph7sg" event={"ID":"9ef7edb4-9565-433d-89d2-0abcbd687401","Type":"ContainerStarted","Data":"dacdc3e722cd82d3b951ac0b96ea44aa9b9dc653520481662c6fa6c1ae464c40"}
May 28 23:04:55.128939 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:55.128907    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n77fp" event={"ID":"0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b","Type":"ContainerStarted","Data":"e17a14fcc25c1015c719d023606e7c5d00ba996f83f63f013a33a3425c82533b"}
May 28 23:04:55.129094 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:55.128993    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n77fp"
May 28 23:04:55.130549 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:55.130492    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5589799f6d-qr8d4" event={"ID":"7d5a9573-d1e5-4894-a60c-2db990e96ed0","Type":"ContainerStarted","Data":"c76034ba79a39f12dfefebbcdd8d1a92811cf7b4386b07777520e44948fc63fb"}
May 28 23:04:55.146215 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:55.146112    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-55985d6dd6-ph7sg" podStartSLOduration=2.146092351 podStartE2EDuration="2.146092351s" podCreationTimestamp="2026-05-28 23:04:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:04:55.14361979 +0000 UTC m=+168.337051044" watchObservedRunningTime="2026-05-28 23:04:55.146092351 +0000 UTC m=+168.339523602"
May 28 23:04:55.161193 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:55.161144    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-n77fp" podStartSLOduration=131.473041099 podStartE2EDuration="2m15.161125394s" podCreationTimestamp="2026-05-28 23:02:40 +0000 UTC" firstStartedPulling="2026-05-28 23:04:49.919729121 +0000 UTC m=+163.113160353" lastFinishedPulling="2026-05-28 23:04:53.607813418 +0000 UTC m=+166.801244648" observedRunningTime="2026-05-28 23:04:55.159493496 +0000 UTC m=+168.352924752" watchObservedRunningTime="2026-05-28 23:04:55.161125394 +0000 UTC m=+168.354556648"
May 28 23:04:55.176911 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:55.176833    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5589799f6d-qr8d4" podStartSLOduration=8.130118219 podStartE2EDuration="12.176816715s" podCreationTimestamp="2026-05-28 23:04:43 +0000 UTC" firstStartedPulling="2026-05-28 23:04:49.900525353 +0000 UTC m=+163.093956583" lastFinishedPulling="2026-05-28 23:04:53.947223838 +0000 UTC m=+167.140655079" observedRunningTime="2026-05-28 23:04:55.17582494 +0000 UTC m=+168.369256192" watchObservedRunningTime="2026-05-28 23:04:55.176816715 +0000 UTC m=+168.370247969"
May 28 23:04:55.486708 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:55.486630    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:04:55.489747 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:55.489716    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v6j6q\""
May 28 23:04:55.497752 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:55.497719    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-47c8w"
May 28 23:04:55.653661 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:55.653624    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-47c8w"]
May 28 23:04:55.658383 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:04:55.658349    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8f16cc_755b_4dc9_a1b5_3c33e7149c77.slice/crio-e13c3a4600e9fe77fd5189eec2b3b9b3a96d86ef54d6f782d90fb9323b6e647e WatchSource:0}: Error finding container e13c3a4600e9fe77fd5189eec2b3b9b3a96d86ef54d6f782d90fb9323b6e647e: Status 404 returned error can't find the container with id e13c3a4600e9fe77fd5189eec2b3b9b3a96d86ef54d6f782d90fb9323b6e647e
May 28 23:04:56.135212 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:56.135174    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-47c8w" event={"ID":"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77","Type":"ContainerStarted","Data":"e13c3a4600e9fe77fd5189eec2b3b9b3a96d86ef54d6f782d90fb9323b6e647e"}
May 28 23:04:59.148299 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:59.148254    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-47c8w" event={"ID":"ba8f16cc-755b-4dc9-a1b5-3c33e7149c77","Type":"ContainerStarted","Data":"32bcfaa6c8cd83d3a202bac3b761b25df93b0805582e1fa693d4892ee94167d6"}
May 28 23:04:59.164225 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:04:59.164165    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-47c8w" podStartSLOduration=136.639818164 podStartE2EDuration="2m19.164145606s" podCreationTimestamp="2026-05-28 23:02:40 +0000 UTC" firstStartedPulling="2026-05-28 23:04:55.660764024 +0000 UTC m=+168.854195463" lastFinishedPulling="2026-05-28 23:04:58.185091663 +0000 UTC m=+171.378522905" observedRunningTime="2026-05-28 23:04:59.163078471 +0000 UTC m=+172.356509722" watchObservedRunningTime="2026-05-28 23:04:59.164145606 +0000 UTC m=+172.357576860"
May 28 23:05:03.341136 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:03.341100    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:05:03.341589 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:03.341148    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:05:03.345943 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:03.345918    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:05:03.790934 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:03.790884    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:05:03.791373 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:03.790949    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:05:03.795855 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:03.795831    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:05:04.170121 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:04.170033    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:05:04.170494 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:04.170474    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:05:04.236732 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:04.236696    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5589799f6d-qr8d4"]
May 28 23:05:05.138358 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:05.138328    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n77fp"
May 28 23:05:14.260843 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:14.260810    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:05:14.264884 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:14.264842    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5bc944fdd7-djthh"
May 28 23:05:31.191582 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.191531    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5589799f6d-qr8d4" podUID="7d5a9573-d1e5-4894-a60c-2db990e96ed0" containerName="console" containerID="cri-o://c76034ba79a39f12dfefebbcdd8d1a92811cf7b4386b07777520e44948fc63fb" gracePeriod=15
May 28 23:05:31.466387 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.466363    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5589799f6d-qr8d4_7d5a9573-d1e5-4894-a60c-2db990e96ed0/console/0.log"
May 28 23:05:31.466523 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.466450    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:05:31.513165 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.513137    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-oauth-config\") pod \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") "
May 28 23:05:31.513318 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.513188    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-service-ca\") pod \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") "
May 28 23:05:31.513318 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.513206    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-config\") pod \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") "
May 28 23:05:31.513318 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.513234    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq95d\" (UniqueName: \"kubernetes.io/projected/7d5a9573-d1e5-4894-a60c-2db990e96ed0-kube-api-access-jq95d\") pod \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") "
May 28 23:05:31.513318 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.513300    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-oauth-serving-cert\") pod \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") "
May 28 23:05:31.513536 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.513350    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-serving-cert\") pod \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\" (UID: \"7d5a9573-d1e5-4894-a60c-2db990e96ed0\") "
May 28 23:05:31.513675 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.513643    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-service-ca" (OuterVolumeSpecName: "service-ca") pod "7d5a9573-d1e5-4894-a60c-2db990e96ed0" (UID: "7d5a9573-d1e5-4894-a60c-2db990e96ed0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:05:31.513675 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.513651    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-config" (OuterVolumeSpecName: "console-config") pod "7d5a9573-d1e5-4894-a60c-2db990e96ed0" (UID: "7d5a9573-d1e5-4894-a60c-2db990e96ed0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:05:31.513819 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.513700    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7d5a9573-d1e5-4894-a60c-2db990e96ed0" (UID: "7d5a9573-d1e5-4894-a60c-2db990e96ed0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:05:31.515674 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.515637    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7d5a9573-d1e5-4894-a60c-2db990e96ed0" (UID: "7d5a9573-d1e5-4894-a60c-2db990e96ed0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:05:31.515674 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.515665    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7d5a9573-d1e5-4894-a60c-2db990e96ed0" (UID: "7d5a9573-d1e5-4894-a60c-2db990e96ed0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:05:31.515830 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.515676    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5a9573-d1e5-4894-a60c-2db990e96ed0-kube-api-access-jq95d" (OuterVolumeSpecName: "kube-api-access-jq95d") pod "7d5a9573-d1e5-4894-a60c-2db990e96ed0" (UID: "7d5a9573-d1e5-4894-a60c-2db990e96ed0"). InnerVolumeSpecName "kube-api-access-jq95d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:05:31.614449 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.614413    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq95d\" (UniqueName: \"kubernetes.io/projected/7d5a9573-d1e5-4894-a60c-2db990e96ed0-kube-api-access-jq95d\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:31.614449 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.614444    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-oauth-serving-cert\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:31.614449 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.614457    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-serving-cert\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:31.614666 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.614471    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-oauth-config\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:31.614666 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.614482    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-service-ca\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:31.614666 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:31.614494    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d5a9573-d1e5-4894-a60c-2db990e96ed0-console-config\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:32.260674 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:32.260643    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5589799f6d-qr8d4_7d5a9573-d1e5-4894-a60c-2db990e96ed0/console/0.log"
May 28 23:05:32.261128 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:32.260695    2577 generic.go:358] "Generic (PLEG): container finished" podID="7d5a9573-d1e5-4894-a60c-2db990e96ed0" containerID="c76034ba79a39f12dfefebbcdd8d1a92811cf7b4386b07777520e44948fc63fb" exitCode=2
May 28 23:05:32.261128 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:32.260760    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5589799f6d-qr8d4"
May 28 23:05:32.261128 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:32.260759    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5589799f6d-qr8d4" event={"ID":"7d5a9573-d1e5-4894-a60c-2db990e96ed0","Type":"ContainerDied","Data":"c76034ba79a39f12dfefebbcdd8d1a92811cf7b4386b07777520e44948fc63fb"}
May 28 23:05:32.261128 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:32.260799    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5589799f6d-qr8d4" event={"ID":"7d5a9573-d1e5-4894-a60c-2db990e96ed0","Type":"ContainerDied","Data":"d1dff1636da6dee6816f3f352d429245d41b5aaf41d7f574af190f31eb58e83f"}
May 28 23:05:32.261128 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:32.260813    2577 scope.go:117] "RemoveContainer" containerID="c76034ba79a39f12dfefebbcdd8d1a92811cf7b4386b07777520e44948fc63fb"
May 28 23:05:32.269719 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:32.269701    2577 scope.go:117] "RemoveContainer" containerID="c76034ba79a39f12dfefebbcdd8d1a92811cf7b4386b07777520e44948fc63fb"
May 28 23:05:32.270077 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:05:32.270056    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c76034ba79a39f12dfefebbcdd8d1a92811cf7b4386b07777520e44948fc63fb\": container with ID starting with c76034ba79a39f12dfefebbcdd8d1a92811cf7b4386b07777520e44948fc63fb not found: ID does not exist" containerID="c76034ba79a39f12dfefebbcdd8d1a92811cf7b4386b07777520e44948fc63fb"
May 28 23:05:32.270141 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:32.270085    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c76034ba79a39f12dfefebbcdd8d1a92811cf7b4386b07777520e44948fc63fb"} err="failed to get container status \"c76034ba79a39f12dfefebbcdd8d1a92811cf7b4386b07777520e44948fc63fb\": rpc error: code = NotFound desc = could not find container \"c76034ba79a39f12dfefebbcdd8d1a92811cf7b4386b07777520e44948fc63fb\": container with ID starting with c76034ba79a39f12dfefebbcdd8d1a92811cf7b4386b07777520e44948fc63fb not found: ID does not exist"
May 28 23:05:32.282692 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:32.282665    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5589799f6d-qr8d4"]
May 28 23:05:32.287656 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:32.287623    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5589799f6d-qr8d4"]
May 28 23:05:33.490768 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:33.490736    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d5a9573-d1e5-4894-a60c-2db990e96ed0" path="/var/lib/kubelet/pods/7d5a9573-d1e5-4894-a60c-2db990e96ed0/volumes"
May 28 23:05:49.768704 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:49.768662    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 23:05:49.769316 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:49.769289    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="alertmanager" containerID="cri-o://6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6" gracePeriod=120
May 28 23:05:49.769407 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:49.769343    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="kube-rbac-proxy-metric" containerID="cri-o://2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e" gracePeriod=120
May 28 23:05:49.769463 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:49.769398    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="kube-rbac-proxy" containerID="cri-o://6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f" gracePeriod=120
May 28 23:05:49.769463 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:49.769350    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="kube-rbac-proxy-web" containerID="cri-o://cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27" gracePeriod=120
May 28 23:05:49.769463 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:49.769417    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="config-reloader" containerID="cri-o://2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e" gracePeriod=120
May 28 23:05:49.769582 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:49.769424    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="prom-label-proxy" containerID="cri-o://d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f" gracePeriod=120
May 28 23:05:50.314397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:50.314367    2577 generic.go:358] "Generic (PLEG): container finished" podID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerID="d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f" exitCode=0
May 28 23:05:50.314397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:50.314394    2577 generic.go:358] "Generic (PLEG): container finished" podID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerID="6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f" exitCode=0
May 28 23:05:50.314397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:50.314401    2577 generic.go:358] "Generic (PLEG): container finished" podID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerID="2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e" exitCode=0
May 28 23:05:50.314397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:50.314407    2577 generic.go:358] "Generic (PLEG): container finished" podID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerID="6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6" exitCode=0
May 28 23:05:50.314697 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:50.314444    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a42853-d4b8-4c40-92ad-c991c665f9bf","Type":"ContainerDied","Data":"d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f"}
May 28 23:05:50.314697 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:50.314483    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a42853-d4b8-4c40-92ad-c991c665f9bf","Type":"ContainerDied","Data":"6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f"}
May 28 23:05:50.314697 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:50.314494    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a42853-d4b8-4c40-92ad-c991c665f9bf","Type":"ContainerDied","Data":"2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e"}
May 28 23:05:50.314697 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:50.314503    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a42853-d4b8-4c40-92ad-c991c665f9bf","Type":"ContainerDied","Data":"6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6"}
May 28 23:05:51.013687 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.013665    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.078848 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.078819    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-main-tls\") pod \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") "
May 28 23:05:51.078848 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.078852    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") "
May 28 23:05:51.079096 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.078888    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3a42853-d4b8-4c40-92ad-c991c665f9bf-config-out\") pod \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") "
May 28 23:05:51.079096 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.079024    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-config-volume\") pod \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") "
May 28 23:05:51.079096 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.079084    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzfqv\" (UniqueName: \"kubernetes.io/projected/f3a42853-d4b8-4c40-92ad-c991c665f9bf-kube-api-access-lzfqv\") pod \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") "
May 28 23:05:51.079246 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.079124    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-cluster-tls-config\") pod \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") "
May 28 23:05:51.079246 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.079153    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy\") pod \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") "
May 28 23:05:51.079246 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.079187    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-web-config\") pod \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") "
May 28 23:05:51.079246 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.079216    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3a42853-d4b8-4c40-92ad-c991c665f9bf-metrics-client-ca\") pod \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") "
May 28 23:05:51.079435 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.079274    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3a42853-d4b8-4c40-92ad-c991c665f9bf-tls-assets\") pod \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") "
May 28 23:05:51.079435 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.079306    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3a42853-d4b8-4c40-92ad-c991c665f9bf-alertmanager-main-db\") pod \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") "
May 28 23:05:51.079435 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.079346    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") "
May 28 23:05:51.079435 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.079379    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3a42853-d4b8-4c40-92ad-c991c665f9bf-alertmanager-trusted-ca-bundle\") pod \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\" (UID: \"f3a42853-d4b8-4c40-92ad-c991c665f9bf\") "
May 28 23:05:51.080309 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.080036    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a42853-d4b8-4c40-92ad-c991c665f9bf-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f3a42853-d4b8-4c40-92ad-c991c665f9bf" (UID: "f3a42853-d4b8-4c40-92ad-c991c665f9bf"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:05:51.080967 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.080844    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a42853-d4b8-4c40-92ad-c991c665f9bf-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f3a42853-d4b8-4c40-92ad-c991c665f9bf" (UID: "f3a42853-d4b8-4c40-92ad-c991c665f9bf"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:05:51.081401 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.081367    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a42853-d4b8-4c40-92ad-c991c665f9bf-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f3a42853-d4b8-4c40-92ad-c991c665f9bf" (UID: "f3a42853-d4b8-4c40-92ad-c991c665f9bf"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:05:51.082923 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.082894    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f3a42853-d4b8-4c40-92ad-c991c665f9bf" (UID: "f3a42853-d4b8-4c40-92ad-c991c665f9bf"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:05:51.083157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.083130    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a42853-d4b8-4c40-92ad-c991c665f9bf-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f3a42853-d4b8-4c40-92ad-c991c665f9bf" (UID: "f3a42853-d4b8-4c40-92ad-c991c665f9bf"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:05:51.083372 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.083264    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a42853-d4b8-4c40-92ad-c991c665f9bf-config-out" (OuterVolumeSpecName: "config-out") pod "f3a42853-d4b8-4c40-92ad-c991c665f9bf" (UID: "f3a42853-d4b8-4c40-92ad-c991c665f9bf"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:05:51.083372 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.083340    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-config-volume" (OuterVolumeSpecName: "config-volume") pod "f3a42853-d4b8-4c40-92ad-c991c665f9bf" (UID: "f3a42853-d4b8-4c40-92ad-c991c665f9bf"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:05:51.083543 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.083515    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f3a42853-d4b8-4c40-92ad-c991c665f9bf" (UID: "f3a42853-d4b8-4c40-92ad-c991c665f9bf"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:05:51.083689 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.083580    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f3a42853-d4b8-4c40-92ad-c991c665f9bf" (UID: "f3a42853-d4b8-4c40-92ad-c991c665f9bf"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:05:51.084139 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.084106    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f3a42853-d4b8-4c40-92ad-c991c665f9bf" (UID: "f3a42853-d4b8-4c40-92ad-c991c665f9bf"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:05:51.085252 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.085230    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a42853-d4b8-4c40-92ad-c991c665f9bf-kube-api-access-lzfqv" (OuterVolumeSpecName: "kube-api-access-lzfqv") pod "f3a42853-d4b8-4c40-92ad-c991c665f9bf" (UID: "f3a42853-d4b8-4c40-92ad-c991c665f9bf"). InnerVolumeSpecName "kube-api-access-lzfqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:05:51.089787 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.089742    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f3a42853-d4b8-4c40-92ad-c991c665f9bf" (UID: "f3a42853-d4b8-4c40-92ad-c991c665f9bf"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:05:51.096577 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.096554    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-web-config" (OuterVolumeSpecName: "web-config") pod "f3a42853-d4b8-4c40-92ad-c991c665f9bf" (UID: "f3a42853-d4b8-4c40-92ad-c991c665f9bf"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:05:51.180212 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.180179    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3a42853-d4b8-4c40-92ad-c991c665f9bf-tls-assets\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:51.180212 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.180209    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3a42853-d4b8-4c40-92ad-c991c665f9bf-alertmanager-main-db\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:51.180394 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.180222    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:51.180394 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.180231    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3a42853-d4b8-4c40-92ad-c991c665f9bf-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:51.180394 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.180242    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-main-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:51.180394 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.180251    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:51.180394 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.180260    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3a42853-d4b8-4c40-92ad-c991c665f9bf-config-out\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:51.180394 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.180268    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-config-volume\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:51.180394 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.180276    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzfqv\" (UniqueName: \"kubernetes.io/projected/f3a42853-d4b8-4c40-92ad-c991c665f9bf-kube-api-access-lzfqv\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:51.180394 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.180284    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-cluster-tls-config\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:51.180394 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.180292    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:51.180394 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.180302    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3a42853-d4b8-4c40-92ad-c991c665f9bf-web-config\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:51.180394 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.180310    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3a42853-d4b8-4c40-92ad-c991c665f9bf-metrics-client-ca\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:05:51.326937 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.326901    2577 generic.go:358] "Generic (PLEG): container finished" podID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerID="2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e" exitCode=0
May 28 23:05:51.326937 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.326929    2577 generic.go:358] "Generic (PLEG): container finished" podID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerID="cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27" exitCode=0
May 28 23:05:51.327130 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.326982    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a42853-d4b8-4c40-92ad-c991c665f9bf","Type":"ContainerDied","Data":"2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e"}
May 28 23:05:51.327130 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.327028    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a42853-d4b8-4c40-92ad-c991c665f9bf","Type":"ContainerDied","Data":"cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27"}
May 28 23:05:51.327130 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.327043    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3a42853-d4b8-4c40-92ad-c991c665f9bf","Type":"ContainerDied","Data":"d7ea680aa5a7d6e36812afce699f3e53553c6d0e1a8d8f8547aef38a88737240"}
May 28 23:05:51.327130 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.327053    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.327130 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.327063    2577 scope.go:117] "RemoveContainer" containerID="d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f"
May 28 23:05:51.335257 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.335239    2577 scope.go:117] "RemoveContainer" containerID="2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e"
May 28 23:05:51.341889 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.341852    2577 scope.go:117] "RemoveContainer" containerID="6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f"
May 28 23:05:51.348561 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.348544    2577 scope.go:117] "RemoveContainer" containerID="cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27"
May 28 23:05:51.351348 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.351328    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 23:05:51.355599 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.355567    2577 scope.go:117] "RemoveContainer" containerID="2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e"
May 28 23:05:51.357860 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.357840    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 23:05:51.362713 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.362695    2577 scope.go:117] "RemoveContainer" containerID="6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6"
May 28 23:05:51.369383 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.369364    2577 scope.go:117] "RemoveContainer" containerID="82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199"
May 28 23:05:51.375934 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.375916    2577 scope.go:117] "RemoveContainer" containerID="d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f"
May 28 23:05:51.376178 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:05:51.376161    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f\": container with ID starting with d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f not found: ID does not exist" containerID="d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f"
May 28 23:05:51.376271 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.376185    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f"} err="failed to get container status \"d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f\": rpc error: code = NotFound desc = could not find container \"d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f\": container with ID starting with d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f not found: ID does not exist"
May 28 23:05:51.376271 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.376203    2577 scope.go:117] "RemoveContainer" containerID="2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e"
May 28 23:05:51.376445 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:05:51.376419    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e\": container with ID starting with 2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e not found: ID does not exist" containerID="2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e"
May 28 23:05:51.376514 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.376452    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e"} err="failed to get container status \"2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e\": rpc error: code = NotFound desc = could not find container \"2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e\": container with ID starting with 2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e not found: ID does not exist"
May 28 23:05:51.376514 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.376476    2577 scope.go:117] "RemoveContainer" containerID="6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f"
May 28 23:05:51.376809 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:05:51.376778    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f\": container with ID starting with 6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f not found: ID does not exist" containerID="6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f"
May 28 23:05:51.377003 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.376816    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f"} err="failed to get container status \"6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f\": rpc error: code = NotFound desc = could not find container \"6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f\": container with ID starting with 6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f not found: ID does not exist"
May 28 23:05:51.377003 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.376836    2577 scope.go:117] "RemoveContainer" containerID="cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27"
May 28 23:05:51.377208 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:05:51.377191    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27\": container with ID starting with cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27 not found: ID does not exist" containerID="cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27"
May 28 23:05:51.377249 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.377213    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27"} err="failed to get container status \"cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27\": rpc error: code = NotFound desc = could not find container \"cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27\": container with ID starting with cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27 not found: ID does not exist"
May 28 23:05:51.377249 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.377230    2577 scope.go:117] "RemoveContainer" containerID="2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e"
May 28 23:05:51.377465 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:05:51.377446    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e\": container with ID starting with 2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e not found: ID does not exist" containerID="2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e"
May 28 23:05:51.377526 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.377473    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e"} err="failed to get container status \"2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e\": rpc error: code = NotFound desc = could not find container \"2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e\": container with ID starting with 2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e not found: ID does not exist"
May 28 23:05:51.377526 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.377489    2577 scope.go:117] "RemoveContainer" containerID="6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6"
May 28 23:05:51.377681 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:05:51.377667    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6\": container with ID starting with 6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6 not found: ID does not exist" containerID="6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6"
May 28 23:05:51.377724 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.377685    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6"} err="failed to get container status \"6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6\": rpc error: code = NotFound desc = could not find container \"6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6\": container with ID starting with 6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6 not found: ID does not exist"
May 28 23:05:51.377724 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.377698    2577 scope.go:117] "RemoveContainer" containerID="82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199"
May 28 23:05:51.377899 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:05:51.377854    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199\": container with ID starting with 82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199 not found: ID does not exist" containerID="82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199"
May 28 23:05:51.377979 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.377908    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199"} err="failed to get container status \"82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199\": rpc error: code = NotFound desc = could not find container \"82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199\": container with ID starting with 82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199 not found: ID does not exist"
May 28 23:05:51.377979 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.377930    2577 scope.go:117] "RemoveContainer" containerID="d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f"
May 28 23:05:51.378170 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.378152    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f"} err="failed to get container status \"d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f\": rpc error: code = NotFound desc = could not find container \"d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f\": container with ID starting with d0dda3faea9756093db24722b9699b49e260a1be9ae5aaa6b5408b259902396f not found: ID does not exist"
May 28 23:05:51.378218 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.378171    2577 scope.go:117] "RemoveContainer" containerID="2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e"
May 28 23:05:51.378386 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.378363    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e"} err="failed to get container status \"2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e\": rpc error: code = NotFound desc = could not find container \"2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e\": container with ID starting with 2857502c2c824bd94741e355c1ca70dcfeba7c5bf0bc3cd1cf02b72e23b7744e not found: ID does not exist"
May 28 23:05:51.378427 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.378389    2577 scope.go:117] "RemoveContainer" containerID="6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f"
May 28 23:05:51.378633 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.378617    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f"} err="failed to get container status \"6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f\": rpc error: code = NotFound desc = could not find container \"6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f\": container with ID starting with 6c12d3e1405abba7b869a7e1560534068393bba1e13e70fb5b301321924cbf4f not found: ID does not exist"
May 28 23:05:51.378633 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.378632    2577 scope.go:117] "RemoveContainer" containerID="cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27"
May 28 23:05:51.378841 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.378826    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27"} err="failed to get container status \"cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27\": rpc error: code = NotFound desc = could not find container \"cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27\": container with ID starting with cb812323defc96c1cadc5c6268e009bd94d574b3f41da83a70c07bb9c559fc27 not found: ID does not exist"
May 28 23:05:51.378841 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.378840    2577 scope.go:117] "RemoveContainer" containerID="2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e"
May 28 23:05:51.379038 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.379022    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e"} err="failed to get container status \"2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e\": rpc error: code = NotFound desc = could not find container \"2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e\": container with ID starting with 2b5ee101e522180a397557128e1545e19f99c8565cefab336546a050de99419e not found: ID does not exist"
May 28 23:05:51.379080 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.379039    2577 scope.go:117] "RemoveContainer" containerID="6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6"
May 28 23:05:51.379258 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.379239    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6"} err="failed to get container status \"6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6\": rpc error: code = NotFound desc = could not find container \"6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6\": container with ID starting with 6384d0b9cf7ea270fb8f959928fc3ff3cee9d7be3ae700dbb2ee6ecda38fefa6 not found: ID does not exist"
May 28 23:05:51.379317 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.379259    2577 scope.go:117] "RemoveContainer" containerID="82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199"
May 28 23:05:51.379465 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.379448    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199"} err="failed to get container status \"82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199\": rpc error: code = NotFound desc = could not find container \"82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199\": container with ID starting with 82354a1c01bb7ef934f77c6907e72bc755bbdc8198c35902620585a200b8b199 not found: ID does not exist"
May 28 23:05:51.386069 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386040    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 23:05:51.386401 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386387    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="config-reloader"
May 28 23:05:51.386462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386406    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="config-reloader"
May 28 23:05:51.386462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386421    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="prom-label-proxy"
May 28 23:05:51.386462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386427    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="prom-label-proxy"
May 28 23:05:51.386462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386435    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="alertmanager"
May 28 23:05:51.386462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386440    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="alertmanager"
May 28 23:05:51.386462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386446    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="kube-rbac-proxy-web"
May 28 23:05:51.386462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386451    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="kube-rbac-proxy-web"
May 28 23:05:51.386462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386458    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d5a9573-d1e5-4894-a60c-2db990e96ed0" containerName="console"
May 28 23:05:51.386462 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386463    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d5a9573-d1e5-4894-a60c-2db990e96ed0" containerName="console"
May 28 23:05:51.386794 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386471    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="kube-rbac-proxy"
May 28 23:05:51.386794 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386480    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="kube-rbac-proxy"
May 28 23:05:51.386794 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386493    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="kube-rbac-proxy-metric"
May 28 23:05:51.386794 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386501    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="kube-rbac-proxy-metric"
May 28 23:05:51.386794 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386512    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="init-config-reloader"
May 28 23:05:51.386794 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386517    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="init-config-reloader"
May 28 23:05:51.386794 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386566    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="alertmanager"
May 28 23:05:51.386794 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386579    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="7d5a9573-d1e5-4894-a60c-2db990e96ed0" containerName="console"
May 28 23:05:51.386794 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386587    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="kube-rbac-proxy-web"
May 28 23:05:51.386794 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386595    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="kube-rbac-proxy"
May 28 23:05:51.386794 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386610    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="prom-label-proxy"
May 28 23:05:51.386794 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386616    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="config-reloader"
May 28 23:05:51.386794 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.386623    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" containerName="kube-rbac-proxy-metric"
May 28 23:05:51.389742 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.389727    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.392467 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.392450    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 28 23:05:51.392798 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.392783    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 28 23:05:51.392928 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.392907    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 28 23:05:51.392995 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.392920    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 28 23:05:51.393083 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.393058    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 28 23:05:51.393083 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.393070    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 28 23:05:51.393344 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.393325    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 28 23:05:51.393525 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.393512    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zqxhv\""
May 28 23:05:51.393525 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.393522    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 28 23:05:51.399725 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.399707    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 28 23:05:51.404046 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.404022    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 23:05:51.483366 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.483333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.483366 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.483369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5537b0f-d674-4098-9fa9-7f3221703da8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.483555 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.483396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5537b0f-d674-4098-9fa9-7f3221703da8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.483555 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.483482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.483555 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.483517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5537b0f-d674-4098-9fa9-7f3221703da8-config-out\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.483555 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.483542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.483675 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.483563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5537b0f-d674-4098-9fa9-7f3221703da8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.483675 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.483593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.483675 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.483617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5537b0f-d674-4098-9fa9-7f3221703da8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.483675 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.483647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.483792 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.483679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-web-config\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.483792 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.483705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljd8t\" (UniqueName: \"kubernetes.io/projected/e5537b0f-d674-4098-9fa9-7f3221703da8-kube-api-access-ljd8t\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.483792 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.483724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.491236 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.491206    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3a42853-d4b8-4c40-92ad-c991c665f9bf" path="/var/lib/kubelet/pods/f3a42853-d4b8-4c40-92ad-c991c665f9bf/volumes"
May 28 23:05:51.584952 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.584913    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.584952 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.584947    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5537b0f-d674-4098-9fa9-7f3221703da8-config-out\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.585128 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.585082    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.585178 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.585130    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5537b0f-d674-4098-9fa9-7f3221703da8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.585341 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.585316    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.585423 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.585355    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5537b0f-d674-4098-9fa9-7f3221703da8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.585423 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.585389    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.585525 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.585456    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-web-config\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.585525 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.585493    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ljd8t\" (UniqueName: \"kubernetes.io/projected/e5537b0f-d674-4098-9fa9-7f3221703da8-kube-api-access-ljd8t\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.585635 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.585552    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.585635 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.585597    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.585635 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.585629    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5537b0f-d674-4098-9fa9-7f3221703da8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.585781 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.585682    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5537b0f-d674-4098-9fa9-7f3221703da8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.585987 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.585855    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5537b0f-d674-4098-9fa9-7f3221703da8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.586614 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.586469    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5537b0f-d674-4098-9fa9-7f3221703da8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.588089 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.588007    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5537b0f-d674-4098-9fa9-7f3221703da8-config-out\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.588467 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.588245    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.588467 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.588392    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.588467 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.588405    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5537b0f-d674-4098-9fa9-7f3221703da8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.588685 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.588531    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.588774 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.588705    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5537b0f-d674-4098-9fa9-7f3221703da8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.588994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.588908    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.588994 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.588938    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-web-config\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.589468 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.589447    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.590298 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.590281    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5537b0f-d674-4098-9fa9-7f3221703da8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.594906 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.594835    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljd8t\" (UniqueName: \"kubernetes.io/projected/e5537b0f-d674-4098-9fa9-7f3221703da8-kube-api-access-ljd8t\") pod \"alertmanager-main-0\" (UID: \"e5537b0f-d674-4098-9fa9-7f3221703da8\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.699057 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.699026    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 23:05:51.837126 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:51.836991    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 23:05:51.840317 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:05:51.840283    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5537b0f_d674_4098_9fa9_7f3221703da8.slice/crio-7edbf2a7ed515af947f58135c05b49d6244cc9c42a50b54316ecc8f7222a2187 WatchSource:0}: Error finding container 7edbf2a7ed515af947f58135c05b49d6244cc9c42a50b54316ecc8f7222a2187: Status 404 returned error can't find the container with id 7edbf2a7ed515af947f58135c05b49d6244cc9c42a50b54316ecc8f7222a2187
May 28 23:05:52.331976 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:52.331937    2577 generic.go:358] "Generic (PLEG): container finished" podID="e5537b0f-d674-4098-9fa9-7f3221703da8" containerID="9c89b23f1a84136c0e179c4acffe6df2acda0aa37cdb58226cb2bd18dae219c5" exitCode=0
May 28 23:05:52.332397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:52.332024    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5537b0f-d674-4098-9fa9-7f3221703da8","Type":"ContainerDied","Data":"9c89b23f1a84136c0e179c4acffe6df2acda0aa37cdb58226cb2bd18dae219c5"}
May 28 23:05:52.332397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:52.332061    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5537b0f-d674-4098-9fa9-7f3221703da8","Type":"ContainerStarted","Data":"7edbf2a7ed515af947f58135c05b49d6244cc9c42a50b54316ecc8f7222a2187"}
May 28 23:05:53.340006 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.339968    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5537b0f-d674-4098-9fa9-7f3221703da8","Type":"ContainerStarted","Data":"485362c9e5fd942a7790b247ac76ac2051998f0b578879639a523e0d035d596e"}
May 28 23:05:53.340006 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.340007    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5537b0f-d674-4098-9fa9-7f3221703da8","Type":"ContainerStarted","Data":"c8e4004bebe3d0ed2a9064893d79afbf8572a5f2cea1981361096656db0e96fa"}
May 28 23:05:53.340396 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.340017    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5537b0f-d674-4098-9fa9-7f3221703da8","Type":"ContainerStarted","Data":"a30e3656f122001d23c17b65ae5845fa20c06a9617d8ea4e977720a15fa96d83"}
May 28 23:05:53.340396 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.340028    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5537b0f-d674-4098-9fa9-7f3221703da8","Type":"ContainerStarted","Data":"499d0565b6c6cbfe35ecb7e3d4bdf43b2edf93928858deea65066b3f8fd69b8d"}
May 28 23:05:53.340396 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.340039    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5537b0f-d674-4098-9fa9-7f3221703da8","Type":"ContainerStarted","Data":"98a088c468619a46fde00fede0a526046c7efb8ab0532e128e5af3b55c521c32"}
May 28 23:05:53.340396 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.340046    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5537b0f-d674-4098-9fa9-7f3221703da8","Type":"ContainerStarted","Data":"174782b012bc6760e2aa072c1c4fdbbc2d63a7f569edf495f42f264c72a76c91"}
May 28 23:05:53.368633 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.368586    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.368571765 podStartE2EDuration="2.368571765s" podCreationTimestamp="2026-05-28 23:05:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:05:53.367355559 +0000 UTC m=+226.560786835" watchObservedRunningTime="2026-05-28 23:05:53.368571765 +0000 UTC m=+226.562003062"
May 28 23:05:53.788839 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.788798    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"]
May 28 23:05:53.791350 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.791322    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:53.794024 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.793993    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 28 23:05:53.794024 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.794010    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 28 23:05:53.794436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.794026    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 28 23:05:53.794436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.793993    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 28 23:05:53.794436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.794357    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sxtdk\""
May 28 23:05:53.794436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.794363    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 28 23:05:53.799521 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.799494    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 28 23:05:53.802856 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.802784    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"]
May 28 23:05:53.906768 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.906728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a45ac38f-34d6-495f-9774-5627bb58bcd6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:53.906768 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.906775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lg8m\" (UniqueName: \"kubernetes.io/projected/a45ac38f-34d6-495f-9774-5627bb58bcd6-kube-api-access-5lg8m\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:53.907051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.906834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45ac38f-34d6-495f-9774-5627bb58bcd6-serving-certs-ca-bundle\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:53.907051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.906859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a45ac38f-34d6-495f-9774-5627bb58bcd6-federate-client-tls\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:53.907051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.906925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a45ac38f-34d6-495f-9774-5627bb58bcd6-telemeter-client-tls\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:53.907051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.906940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a45ac38f-34d6-495f-9774-5627bb58bcd6-metrics-client-ca\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:53.907051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.906972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a45ac38f-34d6-495f-9774-5627bb58bcd6-secret-telemeter-client\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:53.907051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:53.907010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45ac38f-34d6-495f-9774-5627bb58bcd6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.007779 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.007739    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45ac38f-34d6-495f-9774-5627bb58bcd6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.008006 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.007830    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a45ac38f-34d6-495f-9774-5627bb58bcd6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.008006 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.007889    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5lg8m\" (UniqueName: \"kubernetes.io/projected/a45ac38f-34d6-495f-9774-5627bb58bcd6-kube-api-access-5lg8m\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.008006 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.007927    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45ac38f-34d6-495f-9774-5627bb58bcd6-serving-certs-ca-bundle\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.008006 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.007952    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a45ac38f-34d6-495f-9774-5627bb58bcd6-federate-client-tls\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.008006 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.007980    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a45ac38f-34d6-495f-9774-5627bb58bcd6-telemeter-client-tls\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.008006 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.008003    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a45ac38f-34d6-495f-9774-5627bb58bcd6-metrics-client-ca\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.008265 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.008039    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a45ac38f-34d6-495f-9774-5627bb58bcd6-secret-telemeter-client\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.008709 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.008676    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45ac38f-34d6-495f-9774-5627bb58bcd6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.009017 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.008633    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45ac38f-34d6-495f-9774-5627bb58bcd6-serving-certs-ca-bundle\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.009017 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.008894    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a45ac38f-34d6-495f-9774-5627bb58bcd6-metrics-client-ca\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.010633 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.010593    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a45ac38f-34d6-495f-9774-5627bb58bcd6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.010791 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.010771    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a45ac38f-34d6-495f-9774-5627bb58bcd6-telemeter-client-tls\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.011184 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.011167    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a45ac38f-34d6-495f-9774-5627bb58bcd6-federate-client-tls\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.011229 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.011187    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a45ac38f-34d6-495f-9774-5627bb58bcd6-secret-telemeter-client\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.017024 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.017000    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lg8m\" (UniqueName: \"kubernetes.io/projected/a45ac38f-34d6-495f-9774-5627bb58bcd6-kube-api-access-5lg8m\") pod \"telemeter-client-7b97578f6d-kmc5w\" (UID: \"a45ac38f-34d6-495f-9774-5627bb58bcd6\") " pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.104215 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.104144    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"
May 28 23:05:54.233072 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.233041    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b97578f6d-kmc5w"]
May 28 23:05:54.234962 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:05:54.234927    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45ac38f_34d6_495f_9774_5627bb58bcd6.slice/crio-eaa88a98b7649492220f85f47fadc16ed237af9d31aa3b08cc06a1295adb5abc WatchSource:0}: Error finding container eaa88a98b7649492220f85f47fadc16ed237af9d31aa3b08cc06a1295adb5abc: Status 404 returned error can't find the container with id eaa88a98b7649492220f85f47fadc16ed237af9d31aa3b08cc06a1295adb5abc
May 28 23:05:54.345076 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:54.345040    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w" event={"ID":"a45ac38f-34d6-495f-9774-5627bb58bcd6","Type":"ContainerStarted","Data":"eaa88a98b7649492220f85f47fadc16ed237af9d31aa3b08cc06a1295adb5abc"}
May 28 23:05:57.363682 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:57.363641    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w" event={"ID":"a45ac38f-34d6-495f-9774-5627bb58bcd6","Type":"ContainerStarted","Data":"14a09ecbd8cf91463e52e5687463343ef8b6ca5a28f46e9f8e4cc361ecff60c3"}
May 28 23:05:57.363682 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:57.363683    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w" event={"ID":"a45ac38f-34d6-495f-9774-5627bb58bcd6","Type":"ContainerStarted","Data":"29520e7cfb9f8fd3b712c2fa2aea07e26540223309e579aab6e69d2ea4feb3a5"}
May 28 23:05:57.364134 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:57.363694    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w" event={"ID":"a45ac38f-34d6-495f-9774-5627bb58bcd6","Type":"ContainerStarted","Data":"471fd5b5fca002848ef27dd7761213f2556af4bb6591f6d8f07a4d5313cd7722"}
May 28 23:05:57.386437 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:57.386374    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7b97578f6d-kmc5w" podStartSLOduration=2.116501003 podStartE2EDuration="4.386356233s" podCreationTimestamp="2026-05-28 23:05:53 +0000 UTC" firstStartedPulling="2026-05-28 23:05:54.236849879 +0000 UTC m=+227.430281109" lastFinishedPulling="2026-05-28 23:05:56.50670509 +0000 UTC m=+229.700136339" observedRunningTime="2026-05-28 23:05:57.385081851 +0000 UTC m=+230.578513104" watchObservedRunningTime="2026-05-28 23:05:57.386356233 +0000 UTC m=+230.579787486"
May 28 23:05:58.389042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.389002    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-b7d75648f-xn7k8"]
May 28 23:05:58.391842 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.391813    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.402383 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.402354    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b7d75648f-xn7k8"]
May 28 23:05:58.548775 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.548740    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-serving-cert\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.548946 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.548792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-oauth-serving-cert\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.548946 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.548808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mtfh\" (UniqueName: \"kubernetes.io/projected/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-kube-api-access-2mtfh\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.548946 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.548826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-trusted-ca-bundle\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.548946 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.548910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-config\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.548946 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.548942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-oauth-config\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.549105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.549024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-service-ca\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.649663 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.649566    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-serving-cert\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.649663 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.649633    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-oauth-serving-cert\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.649663 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.649658    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2mtfh\" (UniqueName: \"kubernetes.io/projected/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-kube-api-access-2mtfh\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.649951 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.649807    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-trusted-ca-bundle\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.649951 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.649876    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-config\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.649951 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.649916    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-oauth-config\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.650156 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.649988    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-service-ca\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.650610 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.650533    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-config\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.650610 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.650619    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-oauth-serving-cert\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.650610 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.650629    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-service-ca\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.650610 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.650675    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-trusted-ca-bundle\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.652269 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.652238    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-serving-cert\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.652467 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.652447    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-oauth-config\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.657524 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.657508    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mtfh\" (UniqueName: \"kubernetes.io/projected/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-kube-api-access-2mtfh\") pod \"console-b7d75648f-xn7k8\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") " pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.702654 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.702620    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:05:58.832834 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:58.832775    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b7d75648f-xn7k8"]
May 28 23:05:58.835553 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:05:58.835528    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c4e934_862e_4ddf_8a0e_2e9e62538f3f.slice/crio-5086b02f525eb95baf582511cbc138ef71e032de94e9ca610625872a67e11dfe WatchSource:0}: Error finding container 5086b02f525eb95baf582511cbc138ef71e032de94e9ca610625872a67e11dfe: Status 404 returned error can't find the container with id 5086b02f525eb95baf582511cbc138ef71e032de94e9ca610625872a67e11dfe
May 28 23:05:59.371371 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:59.371334    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b7d75648f-xn7k8" event={"ID":"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f","Type":"ContainerStarted","Data":"7e8e545455c64eb22e0967fe2da921e9bb5ba46aced322f11e510ad0aa3f5f9b"}
May 28 23:05:59.371371 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:59.371368    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b7d75648f-xn7k8" event={"ID":"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f","Type":"ContainerStarted","Data":"5086b02f525eb95baf582511cbc138ef71e032de94e9ca610625872a67e11dfe"}
May 28 23:05:59.390299 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:05:59.390257    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-b7d75648f-xn7k8" podStartSLOduration=1.390243526 podStartE2EDuration="1.390243526s" podCreationTimestamp="2026-05-28 23:05:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:05:59.388080318 +0000 UTC m=+232.581511568" watchObservedRunningTime="2026-05-28 23:05:59.390243526 +0000 UTC m=+232.583674775"
May 28 23:06:08.703619 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:08.703575    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:06:08.703619 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:08.703621    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:06:08.709451 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:08.709425    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:06:09.406426 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:09.406395    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:06:09.453978 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:09.453945    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55985d6dd6-ph7sg"]
May 28 23:06:34.475546 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.475441    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-55985d6dd6-ph7sg" podUID="9ef7edb4-9565-433d-89d2-0abcbd687401" containerName="console" containerID="cri-o://4e3d394c8b09eb61d7d8a98fceeab6cea4f7c517fd0385860b65048d1844004c" gracePeriod=15
May 28 23:06:34.729233 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.729167    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-55985d6dd6-ph7sg_9ef7edb4-9565-433d-89d2-0abcbd687401/console/0.log"
May 28 23:06:34.729369 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.729246    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:06:34.865987 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.865952    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef7edb4-9565-433d-89d2-0abcbd687401-console-serving-cert\") pod \"9ef7edb4-9565-433d-89d2-0abcbd687401\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") "
May 28 23:06:34.866163 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.866005    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-trusted-ca-bundle\") pod \"9ef7edb4-9565-433d-89d2-0abcbd687401\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") "
May 28 23:06:34.866163 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.866035    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-console-config\") pod \"9ef7edb4-9565-433d-89d2-0abcbd687401\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") "
May 28 23:06:34.866163 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.866057    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ef7edb4-9565-433d-89d2-0abcbd687401-console-oauth-config\") pod \"9ef7edb4-9565-433d-89d2-0abcbd687401\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") "
May 28 23:06:34.866163 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.866095    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-oauth-serving-cert\") pod \"9ef7edb4-9565-433d-89d2-0abcbd687401\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") "
May 28 23:06:34.866163 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.866124    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vrmh\" (UniqueName: \"kubernetes.io/projected/9ef7edb4-9565-433d-89d2-0abcbd687401-kube-api-access-2vrmh\") pod \"9ef7edb4-9565-433d-89d2-0abcbd687401\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") "
May 28 23:06:34.866163 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.866158    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-service-ca\") pod \"9ef7edb4-9565-433d-89d2-0abcbd687401\" (UID: \"9ef7edb4-9565-433d-89d2-0abcbd687401\") "
May 28 23:06:34.866505 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.866472    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9ef7edb4-9565-433d-89d2-0abcbd687401" (UID: "9ef7edb4-9565-433d-89d2-0abcbd687401"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:06:34.866594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.866535    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9ef7edb4-9565-433d-89d2-0abcbd687401" (UID: "9ef7edb4-9565-433d-89d2-0abcbd687401"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:06:34.866594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.866551    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-console-config" (OuterVolumeSpecName: "console-config") pod "9ef7edb4-9565-433d-89d2-0abcbd687401" (UID: "9ef7edb4-9565-433d-89d2-0abcbd687401"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:06:34.866594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.866552    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-service-ca" (OuterVolumeSpecName: "service-ca") pod "9ef7edb4-9565-433d-89d2-0abcbd687401" (UID: "9ef7edb4-9565-433d-89d2-0abcbd687401"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:06:34.866857 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.866838    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-oauth-serving-cert\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:06:34.866857 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.866856    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-service-ca\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:06:34.866857 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.866892    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-trusted-ca-bundle\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:06:34.867164 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.866907    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ef7edb4-9565-433d-89d2-0abcbd687401-console-config\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:06:34.868402 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.868371    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef7edb4-9565-433d-89d2-0abcbd687401-kube-api-access-2vrmh" (OuterVolumeSpecName: "kube-api-access-2vrmh") pod "9ef7edb4-9565-433d-89d2-0abcbd687401" (UID: "9ef7edb4-9565-433d-89d2-0abcbd687401"). InnerVolumeSpecName "kube-api-access-2vrmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:06:34.868515 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.868403    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef7edb4-9565-433d-89d2-0abcbd687401-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9ef7edb4-9565-433d-89d2-0abcbd687401" (UID: "9ef7edb4-9565-433d-89d2-0abcbd687401"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:06:34.868515 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.868428    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef7edb4-9565-433d-89d2-0abcbd687401-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9ef7edb4-9565-433d-89d2-0abcbd687401" (UID: "9ef7edb4-9565-433d-89d2-0abcbd687401"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:06:34.968218 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.968183    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vrmh\" (UniqueName: \"kubernetes.io/projected/9ef7edb4-9565-433d-89d2-0abcbd687401-kube-api-access-2vrmh\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:06:34.968218 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.968212    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef7edb4-9565-433d-89d2-0abcbd687401-console-serving-cert\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:06:34.968218 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:34.968221    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ef7edb4-9565-433d-89d2-0abcbd687401-console-oauth-config\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:06:35.488822 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:35.488795    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-55985d6dd6-ph7sg_9ef7edb4-9565-433d-89d2-0abcbd687401/console/0.log"
May 28 23:06:35.489196 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:35.488836    2577 generic.go:358] "Generic (PLEG): container finished" podID="9ef7edb4-9565-433d-89d2-0abcbd687401" containerID="4e3d394c8b09eb61d7d8a98fceeab6cea4f7c517fd0385860b65048d1844004c" exitCode=2
May 28 23:06:35.489196 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:35.488936    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55985d6dd6-ph7sg"
May 28 23:06:35.491348 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:35.491323    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55985d6dd6-ph7sg" event={"ID":"9ef7edb4-9565-433d-89d2-0abcbd687401","Type":"ContainerDied","Data":"4e3d394c8b09eb61d7d8a98fceeab6cea4f7c517fd0385860b65048d1844004c"}
May 28 23:06:35.491457 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:35.491353    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55985d6dd6-ph7sg" event={"ID":"9ef7edb4-9565-433d-89d2-0abcbd687401","Type":"ContainerDied","Data":"dacdc3e722cd82d3b951ac0b96ea44aa9b9dc653520481662c6fa6c1ae464c40"}
May 28 23:06:35.491457 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:35.491370    2577 scope.go:117] "RemoveContainer" containerID="4e3d394c8b09eb61d7d8a98fceeab6cea4f7c517fd0385860b65048d1844004c"
May 28 23:06:35.499883 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:35.499836    2577 scope.go:117] "RemoveContainer" containerID="4e3d394c8b09eb61d7d8a98fceeab6cea4f7c517fd0385860b65048d1844004c"
May 28 23:06:35.500195 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:06:35.500166    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3d394c8b09eb61d7d8a98fceeab6cea4f7c517fd0385860b65048d1844004c\": container with ID starting with 4e3d394c8b09eb61d7d8a98fceeab6cea4f7c517fd0385860b65048d1844004c not found: ID does not exist" containerID="4e3d394c8b09eb61d7d8a98fceeab6cea4f7c517fd0385860b65048d1844004c"
May 28 23:06:35.500288 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:35.500206    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3d394c8b09eb61d7d8a98fceeab6cea4f7c517fd0385860b65048d1844004c"} err="failed to get container status \"4e3d394c8b09eb61d7d8a98fceeab6cea4f7c517fd0385860b65048d1844004c\": rpc error: code = NotFound desc = could not find container \"4e3d394c8b09eb61d7d8a98fceeab6cea4f7c517fd0385860b65048d1844004c\": container with ID starting with 4e3d394c8b09eb61d7d8a98fceeab6cea4f7c517fd0385860b65048d1844004c not found: ID does not exist"
May 28 23:06:35.514459 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:35.514436    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55985d6dd6-ph7sg"]
May 28 23:06:35.529471 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:35.529439    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55985d6dd6-ph7sg"]
May 28 23:06:37.492721 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:06:37.492690    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ef7edb4-9565-433d-89d2-0abcbd687401" path="/var/lib/kubelet/pods/9ef7edb4-9565-433d-89d2-0abcbd687401/volumes"
May 28 23:07:07.329751 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:07.329719    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:07:07.332091 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:07.332069    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:07:07.339623 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:07.339578    2577 kubelet.go:1628] "Image garbage collection succeeded"
May 28 23:07:09.479274 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.479237    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-9f686bb86-4qd97"]
May 28 23:07:09.481659 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.479659    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef7edb4-9565-433d-89d2-0abcbd687401" containerName="console"
May 28 23:07:09.481659 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.479674    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef7edb4-9565-433d-89d2-0abcbd687401" containerName="console"
May 28 23:07:09.481659 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.479773    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="9ef7edb4-9565-433d-89d2-0abcbd687401" containerName="console"
May 28 23:07:09.483134 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.483112    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.492783 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.492758    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9f686bb86-4qd97"]
May 28 23:07:09.543954 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.543920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-oauth-serving-cert\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.544141 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.543978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-service-ca\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.544141 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.544013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-trusted-ca-bundle\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.544141 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.544039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-serving-cert\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.544141 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.544071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-oauth-config\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.544141 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.544087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xng74\" (UniqueName: \"kubernetes.io/projected/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-kube-api-access-xng74\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.544411 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.544154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-config\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.644749 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.644715    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-serving-cert\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.644967 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.644760    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-oauth-config\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.644967 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.644778    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xng74\" (UniqueName: \"kubernetes.io/projected/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-kube-api-access-xng74\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.645096 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.644971    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-config\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.645096 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.645015    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-oauth-serving-cert\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.645096 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.645070    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-service-ca\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.645254 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.645100    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-trusted-ca-bundle\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.645810 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.645782    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-oauth-serving-cert\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.645965 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.645778    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-service-ca\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.645965 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.645941    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-config\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.646069 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.646006    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-trusted-ca-bundle\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.647394 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.647370    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-oauth-config\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.647477 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.647419    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-serving-cert\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.652508 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.652489    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xng74\" (UniqueName: \"kubernetes.io/projected/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-kube-api-access-xng74\") pod \"console-9f686bb86-4qd97\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") " pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.793260 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.793226    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:09.925523 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.925498    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9f686bb86-4qd97"]
May 28 23:07:09.928162 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:07:09.928136    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dad404_9b97_4372_b78c_c5f1b2cb9f16.slice/crio-870e243a8a5f822b43d9b9ae73ee53dde77d022d80ab863e9c0459589f88d93e WatchSource:0}: Error finding container 870e243a8a5f822b43d9b9ae73ee53dde77d022d80ab863e9c0459589f88d93e: Status 404 returned error can't find the container with id 870e243a8a5f822b43d9b9ae73ee53dde77d022d80ab863e9c0459589f88d93e
May 28 23:07:09.930105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:09.930085    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 23:07:10.600759 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:10.600723    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f686bb86-4qd97" event={"ID":"c1dad404-9b97-4372-b78c-c5f1b2cb9f16","Type":"ContainerStarted","Data":"6191123308b1fe52eb4c3f650f34339ff4c0dfab309e0cf3b17748b979135824"}
May 28 23:07:10.600759 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:10.600762    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f686bb86-4qd97" event={"ID":"c1dad404-9b97-4372-b78c-c5f1b2cb9f16","Type":"ContainerStarted","Data":"870e243a8a5f822b43d9b9ae73ee53dde77d022d80ab863e9c0459589f88d93e"}
May 28 23:07:10.619520 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:10.619474    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-9f686bb86-4qd97" podStartSLOduration=1.619457654 podStartE2EDuration="1.619457654s" podCreationTimestamp="2026-05-28 23:07:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:07:10.61848695 +0000 UTC m=+303.811918204" watchObservedRunningTime="2026-05-28 23:07:10.619457654 +0000 UTC m=+303.812888907"
May 28 23:07:19.794275 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:19.794233    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:19.794275 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:19.794283    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:19.798974 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:19.798950    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:20.638313 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:20.638283    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:07:20.685008 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:20.684952    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b7d75648f-xn7k8"]
May 28 23:07:45.707116 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:45.707058    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-b7d75648f-xn7k8" podUID="b8c4e934-862e-4ddf-8a0e-2e9e62538f3f" containerName="console" containerID="cri-o://7e8e545455c64eb22e0967fe2da921e9bb5ba46aced322f11e510ad0aa3f5f9b" gracePeriod=15
May 28 23:07:45.948534 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:45.948511    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b7d75648f-xn7k8_b8c4e934-862e-4ddf-8a0e-2e9e62538f3f/console/0.log"
May 28 23:07:45.948667 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:45.948586    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:07:46.065975 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.065944    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-service-ca\") pod \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") "
May 28 23:07:46.066141 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.065998    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-config\") pod \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") "
May 28 23:07:46.066141 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.066027    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-serving-cert\") pod \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") "
May 28 23:07:46.066141 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.066055    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-oauth-config\") pod \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") "
May 28 23:07:46.066141 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.066071    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-trusted-ca-bundle\") pod \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") "
May 28 23:07:46.066141 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.066098    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-oauth-serving-cert\") pod \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") "
May 28 23:07:46.066141 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.066129    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mtfh\" (UniqueName: \"kubernetes.io/projected/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-kube-api-access-2mtfh\") pod \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\" (UID: \"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f\") "
May 28 23:07:46.066463 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.066353    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-service-ca" (OuterVolumeSpecName: "service-ca") pod "b8c4e934-862e-4ddf-8a0e-2e9e62538f3f" (UID: "b8c4e934-862e-4ddf-8a0e-2e9e62538f3f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:07:46.066528 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.066508    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-config" (OuterVolumeSpecName: "console-config") pod "b8c4e934-862e-4ddf-8a0e-2e9e62538f3f" (UID: "b8c4e934-862e-4ddf-8a0e-2e9e62538f3f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:07:46.066779 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.066734    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b8c4e934-862e-4ddf-8a0e-2e9e62538f3f" (UID: "b8c4e934-862e-4ddf-8a0e-2e9e62538f3f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:07:46.067092 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.066826    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b8c4e934-862e-4ddf-8a0e-2e9e62538f3f" (UID: "b8c4e934-862e-4ddf-8a0e-2e9e62538f3f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:07:46.068327 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.068292    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b8c4e934-862e-4ddf-8a0e-2e9e62538f3f" (UID: "b8c4e934-862e-4ddf-8a0e-2e9e62538f3f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:07:46.068420 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.068332    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b8c4e934-862e-4ddf-8a0e-2e9e62538f3f" (UID: "b8c4e934-862e-4ddf-8a0e-2e9e62538f3f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:07:46.068459 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.068419    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-kube-api-access-2mtfh" (OuterVolumeSpecName: "kube-api-access-2mtfh") pod "b8c4e934-862e-4ddf-8a0e-2e9e62538f3f" (UID: "b8c4e934-862e-4ddf-8a0e-2e9e62538f3f"). InnerVolumeSpecName "kube-api-access-2mtfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:07:46.167736 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.167702    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-config\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:07:46.167736 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.167733    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-serving-cert\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:07:46.167736 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.167743    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-console-oauth-config\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:07:46.168032 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.167752    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-trusted-ca-bundle\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:07:46.168032 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.167761    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-oauth-serving-cert\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:07:46.168032 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.167769    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mtfh\" (UniqueName: \"kubernetes.io/projected/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-kube-api-access-2mtfh\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:07:46.168032 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.167778    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f-service-ca\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:07:46.718852 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.718826    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b7d75648f-xn7k8_b8c4e934-862e-4ddf-8a0e-2e9e62538f3f/console/0.log"
May 28 23:07:46.719324 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.718882    2577 generic.go:358] "Generic (PLEG): container finished" podID="b8c4e934-862e-4ddf-8a0e-2e9e62538f3f" containerID="7e8e545455c64eb22e0967fe2da921e9bb5ba46aced322f11e510ad0aa3f5f9b" exitCode=2
May 28 23:07:46.719324 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.718923    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b7d75648f-xn7k8" event={"ID":"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f","Type":"ContainerDied","Data":"7e8e545455c64eb22e0967fe2da921e9bb5ba46aced322f11e510ad0aa3f5f9b"}
May 28 23:07:46.719324 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.718945    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b7d75648f-xn7k8" event={"ID":"b8c4e934-862e-4ddf-8a0e-2e9e62538f3f","Type":"ContainerDied","Data":"5086b02f525eb95baf582511cbc138ef71e032de94e9ca610625872a67e11dfe"}
May 28 23:07:46.719324 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.718960    2577 scope.go:117] "RemoveContainer" containerID="7e8e545455c64eb22e0967fe2da921e9bb5ba46aced322f11e510ad0aa3f5f9b"
May 28 23:07:46.719324 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.718969    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b7d75648f-xn7k8"
May 28 23:07:46.727244 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.727224    2577 scope.go:117] "RemoveContainer" containerID="7e8e545455c64eb22e0967fe2da921e9bb5ba46aced322f11e510ad0aa3f5f9b"
May 28 23:07:46.727467 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:07:46.727450    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e8e545455c64eb22e0967fe2da921e9bb5ba46aced322f11e510ad0aa3f5f9b\": container with ID starting with 7e8e545455c64eb22e0967fe2da921e9bb5ba46aced322f11e510ad0aa3f5f9b not found: ID does not exist" containerID="7e8e545455c64eb22e0967fe2da921e9bb5ba46aced322f11e510ad0aa3f5f9b"
May 28 23:07:46.727513 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.727474    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e8e545455c64eb22e0967fe2da921e9bb5ba46aced322f11e510ad0aa3f5f9b"} err="failed to get container status \"7e8e545455c64eb22e0967fe2da921e9bb5ba46aced322f11e510ad0aa3f5f9b\": rpc error: code = NotFound desc = could not find container \"7e8e545455c64eb22e0967fe2da921e9bb5ba46aced322f11e510ad0aa3f5f9b\": container with ID starting with 7e8e545455c64eb22e0967fe2da921e9bb5ba46aced322f11e510ad0aa3f5f9b not found: ID does not exist"
May 28 23:07:46.740547 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.740526    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b7d75648f-xn7k8"]
May 28 23:07:46.746939 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:46.746913    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b7d75648f-xn7k8"]
May 28 23:07:47.491039 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:07:47.491005    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8c4e934-862e-4ddf-8a0e-2e9e62538f3f" path="/var/lib/kubelet/pods/b8c4e934-862e-4ddf-8a0e-2e9e62538f3f/volumes"
May 28 23:10:56.493438 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:56.493355    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs"]
May 28 23:10:56.493923 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:56.493684    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c4e934-862e-4ddf-8a0e-2e9e62538f3f" containerName="console"
May 28 23:10:56.493923 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:56.493696    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c4e934-862e-4ddf-8a0e-2e9e62538f3f" containerName="console"
May 28 23:10:56.493923 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:56.493749    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="b8c4e934-862e-4ddf-8a0e-2e9e62538f3f" containerName="console"
May 28 23:10:56.496385 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:56.496369    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs"
May 28 23:10:56.499117 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:56.499097    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\""
May 28 23:10:56.499968 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:56.499944    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\""
May 28 23:10:56.499968 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:56.499961    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-sjczc\""
May 28 23:10:56.513654 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:56.513626    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs"]
May 28 23:10:56.595440 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:56.595397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfhn2\" (UniqueName: \"kubernetes.io/projected/55638b90-c9f1-4397-986a-35ef1fa93654-kube-api-access-qfhn2\") pod \"feast-operator-controller-manager-6865775464-kvvvs\" (UID: \"55638b90-c9f1-4397-986a-35ef1fa93654\") " pod="feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs"
May 28 23:10:56.696591 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:56.696552    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qfhn2\" (UniqueName: \"kubernetes.io/projected/55638b90-c9f1-4397-986a-35ef1fa93654-kube-api-access-qfhn2\") pod \"feast-operator-controller-manager-6865775464-kvvvs\" (UID: \"55638b90-c9f1-4397-986a-35ef1fa93654\") " pod="feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs"
May 28 23:10:56.706580 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:56.706545    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfhn2\" (UniqueName: \"kubernetes.io/projected/55638b90-c9f1-4397-986a-35ef1fa93654-kube-api-access-qfhn2\") pod \"feast-operator-controller-manager-6865775464-kvvvs\" (UID: \"55638b90-c9f1-4397-986a-35ef1fa93654\") " pod="feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs"
May 28 23:10:56.807037 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:56.806950    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs"
May 28 23:10:56.934823 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:56.934800    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs"]
May 28 23:10:56.937225 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:10:56.937192    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55638b90_c9f1_4397_986a_35ef1fa93654.slice/crio-fbd0f42280c9b215805d651ae8738dd5dd37f776b126d2c897095471e934082c WatchSource:0}: Error finding container fbd0f42280c9b215805d651ae8738dd5dd37f776b126d2c897095471e934082c: Status 404 returned error can't find the container with id fbd0f42280c9b215805d651ae8738dd5dd37f776b126d2c897095471e934082c
May 28 23:10:57.273278 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:57.273240    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs" event={"ID":"55638b90-c9f1-4397-986a-35ef1fa93654","Type":"ContainerStarted","Data":"fbd0f42280c9b215805d651ae8738dd5dd37f776b126d2c897095471e934082c"}
May 28 23:10:58.810126 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:58.810092    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c6874b6d6-wvdxl"]
May 28 23:10:58.820786 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:58.820755    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:58.825724 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:58.825687    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6874b6d6-wvdxl"]
May 28 23:10:58.916220 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:58.916180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94352028-175a-4d33-836e-f99b7397441a-console-oauth-config\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:58.916410 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:58.916271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94352028-175a-4d33-836e-f99b7397441a-oauth-serving-cert\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:58.916410 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:58.916307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94352028-175a-4d33-836e-f99b7397441a-service-ca\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:58.916410 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:58.916359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm2vw\" (UniqueName: \"kubernetes.io/projected/94352028-175a-4d33-836e-f99b7397441a-kube-api-access-zm2vw\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:58.916410 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:58.916394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94352028-175a-4d33-836e-f99b7397441a-console-config\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:58.916584 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:58.916499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94352028-175a-4d33-836e-f99b7397441a-console-serving-cert\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:58.916584 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:58.916534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94352028-175a-4d33-836e-f99b7397441a-trusted-ca-bundle\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.017322 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.017286    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94352028-175a-4d33-836e-f99b7397441a-oauth-serving-cert\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.017498 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.017339    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94352028-175a-4d33-836e-f99b7397441a-service-ca\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.017498 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.017373    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zm2vw\" (UniqueName: \"kubernetes.io/projected/94352028-175a-4d33-836e-f99b7397441a-kube-api-access-zm2vw\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.017498 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.017398    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94352028-175a-4d33-836e-f99b7397441a-console-config\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.017498 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.017444    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94352028-175a-4d33-836e-f99b7397441a-console-serving-cert\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.017498 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.017460    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94352028-175a-4d33-836e-f99b7397441a-trusted-ca-bundle\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.017498 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.017492    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94352028-175a-4d33-836e-f99b7397441a-console-oauth-config\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.018115 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.018084    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94352028-175a-4d33-836e-f99b7397441a-oauth-serving-cert\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.018265 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.018131    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94352028-175a-4d33-836e-f99b7397441a-service-ca\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.018265 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.018246    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94352028-175a-4d33-836e-f99b7397441a-console-config\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.019181 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.019153    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94352028-175a-4d33-836e-f99b7397441a-trusted-ca-bundle\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.020527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.020492    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94352028-175a-4d33-836e-f99b7397441a-console-oauth-config\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.020780 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.020756    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94352028-175a-4d33-836e-f99b7397441a-console-serving-cert\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.026177 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.026155    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm2vw\" (UniqueName: \"kubernetes.io/projected/94352028-175a-4d33-836e-f99b7397441a-kube-api-access-zm2vw\") pod \"console-7c6874b6d6-wvdxl\" (UID: \"94352028-175a-4d33-836e-f99b7397441a\") " pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.138131 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.138055    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:10:59.712349 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:10:59.712322    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6874b6d6-wvdxl"]
May 28 23:10:59.713829 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:10:59.713804    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94352028_175a_4d33_836e_f99b7397441a.slice/crio-fbe46372fd9c7aa330e96c72013d55dd44cbafcfdba50b4195a0f406f5d5215e WatchSource:0}: Error finding container fbe46372fd9c7aa330e96c72013d55dd44cbafcfdba50b4195a0f406f5d5215e: Status 404 returned error can't find the container with id fbe46372fd9c7aa330e96c72013d55dd44cbafcfdba50b4195a0f406f5d5215e
May 28 23:11:00.285499 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:00.285461    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs" event={"ID":"55638b90-c9f1-4397-986a-35ef1fa93654","Type":"ContainerStarted","Data":"0335c6314bf28f311ad7f9faf0ef791ab9a601cfedc88e8cfd7cdab6cd2c93b4"}
May 28 23:11:00.285978 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:00.285526    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs"
May 28 23:11:00.286857 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:00.286833    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6874b6d6-wvdxl" event={"ID":"94352028-175a-4d33-836e-f99b7397441a","Type":"ContainerStarted","Data":"23d95fbbd282e2de0ac28dc0022caf918959d086dd06fe80021e9a17147950ea"}
May 28 23:11:00.286857 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:00.286859    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6874b6d6-wvdxl" event={"ID":"94352028-175a-4d33-836e-f99b7397441a","Type":"ContainerStarted","Data":"fbe46372fd9c7aa330e96c72013d55dd44cbafcfdba50b4195a0f406f5d5215e"}
May 28 23:11:00.301156 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:00.301103    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs" podStartSLOduration=1.599037322 podStartE2EDuration="4.301086487s" podCreationTimestamp="2026-05-28 23:10:56 +0000 UTC" firstStartedPulling="2026-05-28 23:10:56.939227036 +0000 UTC m=+530.132658267" lastFinishedPulling="2026-05-28 23:10:59.641276202 +0000 UTC m=+532.834707432" observedRunningTime="2026-05-28 23:11:00.300568501 +0000 UTC m=+533.493999753" watchObservedRunningTime="2026-05-28 23:11:00.301086487 +0000 UTC m=+533.494517741"
May 28 23:11:00.316487 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:00.316438    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7c6874b6d6-wvdxl" podStartSLOduration=2.316423828 podStartE2EDuration="2.316423828s" podCreationTimestamp="2026-05-28 23:10:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:11:00.315962962 +0000 UTC m=+533.509394214" watchObservedRunningTime="2026-05-28 23:11:00.316423828 +0000 UTC m=+533.509855080"
May 28 23:11:09.138504 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:09.138451    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:11:09.138939 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:09.138600    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:11:09.143444 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:09.143423    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:11:09.318170 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:09.318144    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c6874b6d6-wvdxl"
May 28 23:11:09.368804 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:09.368773    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9f686bb86-4qd97"]
May 28 23:11:11.292011 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:11.291977    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs"
May 28 23:11:34.393910 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.393840    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-9f686bb86-4qd97" podUID="c1dad404-9b97-4372-b78c-c5f1b2cb9f16" containerName="console" containerID="cri-o://6191123308b1fe52eb4c3f650f34339ff4c0dfab309e0cf3b17748b979135824" gracePeriod=15
May 28 23:11:34.639566 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.639539    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9f686bb86-4qd97_c1dad404-9b97-4372-b78c-c5f1b2cb9f16/console/0.log"
May 28 23:11:34.639691 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.639599    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:11:34.719564 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.719490    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-config\") pod \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") "
May 28 23:11:34.719564 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.719528    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-trusted-ca-bundle\") pod \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") "
May 28 23:11:34.719564 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.719550    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-service-ca\") pod \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") "
May 28 23:11:34.719818 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.719575    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-serving-cert\") pod \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") "
May 28 23:11:34.719818 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.719671    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-oauth-config\") pod \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") "
May 28 23:11:34.719818 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.719693    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xng74\" (UniqueName: \"kubernetes.io/projected/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-kube-api-access-xng74\") pod \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") "
May 28 23:11:34.719818 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.719731    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-oauth-serving-cert\") pod \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\" (UID: \"c1dad404-9b97-4372-b78c-c5f1b2cb9f16\") "
May 28 23:11:34.720036 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.719960    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-config" (OuterVolumeSpecName: "console-config") pod "c1dad404-9b97-4372-b78c-c5f1b2cb9f16" (UID: "c1dad404-9b97-4372-b78c-c5f1b2cb9f16"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:11:34.720036 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.719955    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c1dad404-9b97-4372-b78c-c5f1b2cb9f16" (UID: "c1dad404-9b97-4372-b78c-c5f1b2cb9f16"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:11:34.720159 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.720132    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-service-ca" (OuterVolumeSpecName: "service-ca") pod "c1dad404-9b97-4372-b78c-c5f1b2cb9f16" (UID: "c1dad404-9b97-4372-b78c-c5f1b2cb9f16"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:11:34.720223 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.720201    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c1dad404-9b97-4372-b78c-c5f1b2cb9f16" (UID: "c1dad404-9b97-4372-b78c-c5f1b2cb9f16"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:11:34.721725 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.721705    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c1dad404-9b97-4372-b78c-c5f1b2cb9f16" (UID: "c1dad404-9b97-4372-b78c-c5f1b2cb9f16"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:11:34.721813 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.721774    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c1dad404-9b97-4372-b78c-c5f1b2cb9f16" (UID: "c1dad404-9b97-4372-b78c-c5f1b2cb9f16"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:11:34.721991 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.721972    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-kube-api-access-xng74" (OuterVolumeSpecName: "kube-api-access-xng74") pod "c1dad404-9b97-4372-b78c-c5f1b2cb9f16" (UID: "c1dad404-9b97-4372-b78c-c5f1b2cb9f16"). InnerVolumeSpecName "kube-api-access-xng74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:11:34.820468 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.820421    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-oauth-config\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:11:34.820468 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.820461    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xng74\" (UniqueName: \"kubernetes.io/projected/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-kube-api-access-xng74\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:11:34.820468 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.820471    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-oauth-serving-cert\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:11:34.820468 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.820480    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-config\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:11:34.820722 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.820489    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-trusted-ca-bundle\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:11:34.820722 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.820501    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-service-ca\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:11:34.820722 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:34.820511    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1dad404-9b97-4372-b78c-c5f1b2cb9f16-console-serving-cert\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:11:35.399651 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:35.399622    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9f686bb86-4qd97_c1dad404-9b97-4372-b78c-c5f1b2cb9f16/console/0.log"
May 28 23:11:35.400041 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:35.399661    2577 generic.go:358] "Generic (PLEG): container finished" podID="c1dad404-9b97-4372-b78c-c5f1b2cb9f16" containerID="6191123308b1fe52eb4c3f650f34339ff4c0dfab309e0cf3b17748b979135824" exitCode=2
May 28 23:11:35.400041 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:35.399729    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f686bb86-4qd97"
May 28 23:11:35.400041 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:35.399763    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f686bb86-4qd97" event={"ID":"c1dad404-9b97-4372-b78c-c5f1b2cb9f16","Type":"ContainerDied","Data":"6191123308b1fe52eb4c3f650f34339ff4c0dfab309e0cf3b17748b979135824"}
May 28 23:11:35.400041 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:35.399803    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f686bb86-4qd97" event={"ID":"c1dad404-9b97-4372-b78c-c5f1b2cb9f16","Type":"ContainerDied","Data":"870e243a8a5f822b43d9b9ae73ee53dde77d022d80ab863e9c0459589f88d93e"}
May 28 23:11:35.400041 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:35.399818    2577 scope.go:117] "RemoveContainer" containerID="6191123308b1fe52eb4c3f650f34339ff4c0dfab309e0cf3b17748b979135824"
May 28 23:11:35.408479 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:35.408463    2577 scope.go:117] "RemoveContainer" containerID="6191123308b1fe52eb4c3f650f34339ff4c0dfab309e0cf3b17748b979135824"
May 28 23:11:35.408752 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:11:35.408732    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6191123308b1fe52eb4c3f650f34339ff4c0dfab309e0cf3b17748b979135824\": container with ID starting with 6191123308b1fe52eb4c3f650f34339ff4c0dfab309e0cf3b17748b979135824 not found: ID does not exist" containerID="6191123308b1fe52eb4c3f650f34339ff4c0dfab309e0cf3b17748b979135824"
May 28 23:11:35.408807 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:35.408764    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6191123308b1fe52eb4c3f650f34339ff4c0dfab309e0cf3b17748b979135824"} err="failed to get container status \"6191123308b1fe52eb4c3f650f34339ff4c0dfab309e0cf3b17748b979135824\": rpc error: code = NotFound desc = could not find container \"6191123308b1fe52eb4c3f650f34339ff4c0dfab309e0cf3b17748b979135824\": container with ID starting with 6191123308b1fe52eb4c3f650f34339ff4c0dfab309e0cf3b17748b979135824 not found: ID does not exist"
May 28 23:11:35.420922 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:35.420895    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9f686bb86-4qd97"]
May 28 23:11:35.424969 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:35.424948    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9f686bb86-4qd97"]
May 28 23:11:35.491491 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:35.491457    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1dad404-9b97-4372-b78c-c5f1b2cb9f16" path="/var/lib/kubelet/pods/c1dad404-9b97-4372-b78c-c5f1b2cb9f16/volumes"
May 28 23:11:39.455480 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.455441    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"]
May 28 23:11:39.455957 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.455821    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1dad404-9b97-4372-b78c-c5f1b2cb9f16" containerName="console"
May 28 23:11:39.455957 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.455838    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1dad404-9b97-4372-b78c-c5f1b2cb9f16" containerName="console"
May 28 23:11:39.455957 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.455941    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="c1dad404-9b97-4372-b78c-c5f1b2cb9f16" containerName="console"
May 28 23:11:39.461841 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.461811    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:39.464588 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.464560    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 23:11:39.464767 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.464739    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 28 23:11:39.464860 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.464845    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-2jzb6\""
May 28 23:11:39.465097 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.465076    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 28 23:11:39.465151 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.465095    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 28 23:11:39.465257 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.465244    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 28 23:11:39.465326 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.465287    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 23:11:39.465405 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.465393    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 28 23:11:39.472912 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.472889    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"]
May 28 23:11:39.563521 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.563483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-online-tls\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:39.563712 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.563531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-registry-tls\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:39.563712 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.563561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-offline-tls\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:39.563712 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.563623    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs9s8\" (UniqueName: \"kubernetes.io/projected/5da4d307-d559-49f3-b2e6-8c4da71e2c96-kube-api-access-fs9s8\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:39.563712 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.563694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5da4d307-d559-49f3-b2e6-8c4da71e2c96-feast-data\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:39.563936 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.563733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-ui-tls\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:39.664999 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.664949    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-ui-tls\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:39.665185 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.665041    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-online-tls\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:39.665185 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.665071    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-registry-tls\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:39.665185 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.665095    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-offline-tls\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:39.665185 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.665162    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fs9s8\" (UniqueName: \"kubernetes.io/projected/5da4d307-d559-49f3-b2e6-8c4da71e2c96-kube-api-access-fs9s8\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:39.665381 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:11:39.665192    2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 28 23:11:39.665381 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:11:39.665205    2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 28 23:11:39.665381 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.665200    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5da4d307-d559-49f3-b2e6-8c4da71e2c96-feast-data\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:39.665381 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:11:39.665278    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-registry-tls podName:5da4d307-d559-49f3-b2e6-8c4da71e2c96 nodeName:}" failed. No retries permitted until 2026-05-28 23:11:40.165256385 +0000 UTC m=+573.358687626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-registry-tls") pod "feast-simple-feast-setup-f8c865756-kv9px" (UID: "5da4d307-d559-49f3-b2e6-8c4da71e2c96") : secret "feast-simple-feast-setup-registry-tls" not found
May 28 23:11:39.665381 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:11:39.665323    2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 28 23:11:39.665381 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:11:39.665368    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-online-tls podName:5da4d307-d559-49f3-b2e6-8c4da71e2c96 nodeName:}" failed. No retries permitted until 2026-05-28 23:11:40.165350057 +0000 UTC m=+573.358781301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-online-tls") pod "feast-simple-feast-setup-f8c865756-kv9px" (UID: "5da4d307-d559-49f3-b2e6-8c4da71e2c96") : secret "feast-simple-feast-setup-online-tls" not found
May 28 23:11:39.665678 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:11:39.665386    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-offline-tls podName:5da4d307-d559-49f3-b2e6-8c4da71e2c96 nodeName:}" failed. No retries permitted until 2026-05-28 23:11:40.165377026 +0000 UTC m=+573.358808271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-offline-tls") pod "feast-simple-feast-setup-f8c865756-kv9px" (UID: "5da4d307-d559-49f3-b2e6-8c4da71e2c96") : secret "feast-simple-feast-setup-offline-tls" not found
May 28 23:11:39.665678 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.665541    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5da4d307-d559-49f3-b2e6-8c4da71e2c96-feast-data\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:39.667976 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.667947    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-ui-tls\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:39.676829 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:39.676803    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs9s8\" (UniqueName: \"kubernetes.io/projected/5da4d307-d559-49f3-b2e6-8c4da71e2c96-kube-api-access-fs9s8\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:40.169447 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:40.169410    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-online-tls\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:40.169447 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:40.169451    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-registry-tls\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:40.169695 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:40.169472    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-offline-tls\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:40.172159 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:40.172124    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-registry-tls\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:40.172278 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:40.172175    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-online-tls\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:40.172278 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:40.172181    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-offline-tls\") pod \"feast-simple-feast-setup-f8c865756-kv9px\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:40.386250 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:40.386215    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:11:40.526068 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:40.526032    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"]
May 28 23:11:40.527968 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:11:40.527937    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5da4d307_d559_49f3_b2e6_8c4da71e2c96.slice/crio-6b09d34f46da0f69a9aa13b04fd96b5f709b546f8d4d3bae3450750db4315b3e WatchSource:0}: Error finding container 6b09d34f46da0f69a9aa13b04fd96b5f709b546f8d4d3bae3450750db4315b3e: Status 404 returned error can't find the container with id 6b09d34f46da0f69a9aa13b04fd96b5f709b546f8d4d3bae3450750db4315b3e
May 28 23:11:41.424562 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:11:41.424511    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" event={"ID":"5da4d307-d559-49f3-b2e6-8c4da71e2c96","Type":"ContainerStarted","Data":"6b09d34f46da0f69a9aa13b04fd96b5f709b546f8d4d3bae3450750db4315b3e"}
May 28 23:12:12.959104 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:12.959068    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:12:12.960193 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:12.960162    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:12:13.539571 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:13.539530    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" event={"ID":"5da4d307-d559-49f3-b2e6-8c4da71e2c96","Type":"ContainerStarted","Data":"9ffe729a419929e59ddc180e2e51c94b35c5c162c913a924202afd3bacad14ac"}
May 28 23:12:18.558327 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:18.558286    2577 generic.go:358] "Generic (PLEG): container finished" podID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerID="9ffe729a419929e59ddc180e2e51c94b35c5c162c913a924202afd3bacad14ac" exitCode=0
May 28 23:12:18.558736 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:18.558357    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" event={"ID":"5da4d307-d559-49f3-b2e6-8c4da71e2c96","Type":"ContainerDied","Data":"9ffe729a419929e59ddc180e2e51c94b35c5c162c913a924202afd3bacad14ac"}
May 28 23:12:18.566936 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:18.566915    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 23:12:19.564033 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:19.563996    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" event={"ID":"5da4d307-d559-49f3-b2e6-8c4da71e2c96","Type":"ContainerStarted","Data":"c6504d6412efd70444cbab2bbe9ad72fe5a62e795c611a76c93c5faadec50a89"}
May 28 23:12:24.586913 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:24.586782    2577 generic.go:358] "Generic (PLEG): container finished" podID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerID="c6504d6412efd70444cbab2bbe9ad72fe5a62e795c611a76c93c5faadec50a89" exitCode=0
May 28 23:12:24.586913 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:24.586860    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" event={"ID":"5da4d307-d559-49f3-b2e6-8c4da71e2c96","Type":"ContainerDied","Data":"c6504d6412efd70444cbab2bbe9ad72fe5a62e795c611a76c93c5faadec50a89"}
May 28 23:12:24.597351 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:24.597330    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 28 23:12:25.595559 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:25.595512    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" event={"ID":"5da4d307-d559-49f3-b2e6-8c4da71e2c96","Type":"ContainerStarted","Data":"4565e0a62c3d4567d2fe36b142214523c616a136eb9e8985640b8dcc8b4238dc"}
May 28 23:12:25.596062 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:25.595570    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" event={"ID":"5da4d307-d559-49f3-b2e6-8c4da71e2c96","Type":"ContainerStarted","Data":"4b87909cc860623fd7062f5c9432bf6f653f165b0d79a0dbf72af86ad872cab1"}
May 28 23:12:25.596062 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:25.595586    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" event={"ID":"5da4d307-d559-49f3-b2e6-8c4da71e2c96","Type":"ContainerStarted","Data":"edcc991f03d8ba26a49bdf1f2e05705d0dcfd4696e9f77222a8f22a8c41530d1"}
May 28 23:12:25.596062 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:25.595599    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" event={"ID":"5da4d307-d559-49f3-b2e6-8c4da71e2c96","Type":"ContainerStarted","Data":"141b400a99ef6b11a5be2dff63a17946144852876d2cfed601f8b2e27ec56ad9"}
May 28 23:12:25.621400 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:25.621330    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" podStartSLOduration=14.097481233 podStartE2EDuration="46.621312004s" podCreationTimestamp="2026-05-28 23:11:39 +0000 UTC" firstStartedPulling="2026-05-28 23:11:40.529848896 +0000 UTC m=+573.723280127" lastFinishedPulling="2026-05-28 23:12:13.053679664 +0000 UTC m=+606.247110898" observedRunningTime="2026-05-28 23:12:25.620141376 +0000 UTC m=+618.813572652" watchObservedRunningTime="2026-05-28 23:12:25.621312004 +0000 UTC m=+618.814743270"
May 28 23:12:28.386857 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:28.386639    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:12:28.386857 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:28.386688    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:12:28.386857 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:28.386702    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:12:28.386857 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:28.386830    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:12:28.388954 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:28.388894    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.31:8816: connect: connection refused"
May 28 23:12:28.389107 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:28.388894    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.31:8443: connect: connection refused"
May 28 23:12:28.389107 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:28.389011    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused"
May 28 23:12:28.389107 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:28.388896    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused"
May 28 23:12:30.387032 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:30.386989    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:12:30.387459 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:30.387075    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:12:30.387459 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:30.387105    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:12:30.387459 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:30.387114    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:12:31.387783 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:31.387747    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:12:31.388244 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:31.387954    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:12:31.388244 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:31.388145    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:12:31.392318 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:31.392299    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:12:31.620445 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:31.620415    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:12:31.620617 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:31.620485    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:12:31.620709 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:31.620692    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:12:31.622815 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:12:31.622797    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:13:32.698368 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:32.698330    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"]
May 28 23:13:32.698858 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:32.698805    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="registry" containerID="cri-o://141b400a99ef6b11a5be2dff63a17946144852876d2cfed601f8b2e27ec56ad9" gracePeriod=30
May 28 23:13:32.698964 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:32.698888    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="online" containerID="cri-o://edcc991f03d8ba26a49bdf1f2e05705d0dcfd4696e9f77222a8f22a8c41530d1" gracePeriod=30
May 28 23:13:32.698964 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:32.698880    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="ui" containerID="cri-o://4565e0a62c3d4567d2fe36b142214523c616a136eb9e8985640b8dcc8b4238dc" gracePeriod=30
May 28 23:13:32.698964 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:32.698841    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="offline" containerID="cri-o://4b87909cc860623fd7062f5c9432bf6f653f165b0d79a0dbf72af86ad872cab1" gracePeriod=30
May 28 23:13:33.055230 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.055175    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"]
May 28 23:13:33.060081 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.060053    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.068685 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.068655    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"]
May 28 23:13:33.069276 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.069260    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-fcg7f\""
May 28 23:13:33.078564 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.078539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-feast-data\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.078786 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.078768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-registry-tls\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.078919 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.078904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvm7\" (UniqueName: \"kubernetes.io/projected/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-kube-api-access-bxvm7\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.079038 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.079025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-online-tls\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.079187 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.079175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-offline-tls\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.079288 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.079276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-ui-tls\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.180149 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.180111    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-ui-tls\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.180149 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.180167    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-feast-data\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.180422 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.180206    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-registry-tls\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.180422 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.180236    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvm7\" (UniqueName: \"kubernetes.io/projected/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-kube-api-access-bxvm7\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.180422 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.180269    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-online-tls\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.180422 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:13:33.180271    2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 28 23:13:33.180422 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:13:33.180413    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-ui-tls podName:08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e nodeName:}" failed. No retries permitted until 2026-05-28 23:13:33.680391704 +0000 UTC m=+686.873822936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-ui-tls") pod "feast-simple-feast-setup-f8c865756-ftlbf" (UID: "08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e") : secret "feast-simple-feast-setup-ui-tls" not found
May 28 23:13:33.180716 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.180684    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-feast-data\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.180785 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.180745    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-offline-tls\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.180885 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:13:33.180322    2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 28 23:13:33.180989 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:13:33.180965    2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 28 23:13:33.181179 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:13:33.181069    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-registry-tls podName:08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e nodeName:}" failed. No retries permitted until 2026-05-28 23:13:33.681045125 +0000 UTC m=+686.874476369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-registry-tls") pod "feast-simple-feast-setup-f8c865756-ftlbf" (UID: "08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e") : secret "feast-simple-feast-setup-registry-tls" not found
May 28 23:13:33.181258 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:13:33.181218    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-online-tls podName:08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e nodeName:}" failed. No retries permitted until 2026-05-28 23:13:33.681199413 +0000 UTC m=+686.874630648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-online-tls") pod "feast-simple-feast-setup-f8c865756-ftlbf" (UID: "08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e") : secret "feast-simple-feast-setup-online-tls" not found
May 28 23:13:33.183914 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.183889    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-offline-tls\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.193253 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.193225    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvm7\" (UniqueName: \"kubernetes.io/projected/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-kube-api-access-bxvm7\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.685518 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.685429    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-ui-tls\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.685518 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.685498    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-registry-tls\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.685708 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.685525    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-online-tls\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.688037 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.688011    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-online-tls\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.688204 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.688181    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-registry-tls\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.688247 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.688181    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-ui-tls\") pod \"feast-simple-feast-setup-f8c865756-ftlbf\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") " pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:33.839035 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.839003    2577 generic.go:358] "Generic (PLEG): container finished" podID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerID="4565e0a62c3d4567d2fe36b142214523c616a136eb9e8985640b8dcc8b4238dc" exitCode=0
May 28 23:13:33.839035 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.839030    2577 generic.go:358] "Generic (PLEG): container finished" podID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerID="edcc991f03d8ba26a49bdf1f2e05705d0dcfd4696e9f77222a8f22a8c41530d1" exitCode=0
May 28 23:13:33.839438 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.839092    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" event={"ID":"5da4d307-d559-49f3-b2e6-8c4da71e2c96","Type":"ContainerDied","Data":"4565e0a62c3d4567d2fe36b142214523c616a136eb9e8985640b8dcc8b4238dc"}
May 28 23:13:33.839438 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.839131    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" event={"ID":"5da4d307-d559-49f3-b2e6-8c4da71e2c96","Type":"ContainerDied","Data":"edcc991f03d8ba26a49bdf1f2e05705d0dcfd4696e9f77222a8f22a8c41530d1"}
May 28 23:13:33.980424 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:33.980292    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:34.114695 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:34.114666    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"]
May 28 23:13:34.116627 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:13:34.116596    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08b2f6b1_a7f1_49a7_a64d_3235b4be0d1e.slice/crio-026248f1d6eaf8c6636a2e7e10212770a0e09d7f6f5f9d0c5fdd34b75bd6a936 WatchSource:0}: Error finding container 026248f1d6eaf8c6636a2e7e10212770a0e09d7f6f5f9d0c5fdd34b75bd6a936: Status 404 returned error can't find the container with id 026248f1d6eaf8c6636a2e7e10212770a0e09d7f6f5f9d0c5fdd34b75bd6a936
May 28 23:13:34.846371 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:34.846330    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" event={"ID":"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e","Type":"ContainerStarted","Data":"23e3d1ef4e8408603617a2482eeed5af37a5b9b74038684e60d50c4209a1a104"}
May 28 23:13:34.846371 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:34.846369    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" event={"ID":"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e","Type":"ContainerStarted","Data":"026248f1d6eaf8c6636a2e7e10212770a0e09d7f6f5f9d0c5fdd34b75bd6a936"}
May 28 23:13:38.864421 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:38.864384    2577 generic.go:358] "Generic (PLEG): container finished" podID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerID="23e3d1ef4e8408603617a2482eeed5af37a5b9b74038684e60d50c4209a1a104" exitCode=0
May 28 23:13:38.864421 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:38.864425    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" event={"ID":"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e","Type":"ContainerDied","Data":"23e3d1ef4e8408603617a2482eeed5af37a5b9b74038684e60d50c4209a1a104"}
May 28 23:13:39.869784 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:39.869747    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" event={"ID":"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e","Type":"ContainerStarted","Data":"7a00017724a4b1813c37304bd7d47da8a7245ffa0f795e48dfb00f4a0eff823b"}
May 28 23:13:41.620834 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:41.620785    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.31:8443: connect: connection refused"
May 28 23:13:41.621256 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:41.620799    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused"
May 28 23:13:44.887724 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:44.887687    2577 generic.go:358] "Generic (PLEG): container finished" podID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerID="7a00017724a4b1813c37304bd7d47da8a7245ffa0f795e48dfb00f4a0eff823b" exitCode=0
May 28 23:13:44.888142 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:44.887745    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" event={"ID":"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e","Type":"ContainerDied","Data":"7a00017724a4b1813c37304bd7d47da8a7245ffa0f795e48dfb00f4a0eff823b"}
May 28 23:13:45.895262 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:45.895216    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" event={"ID":"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e","Type":"ContainerStarted","Data":"8a065089d0d76b7850aaab807cb4f8de9c56386560dbac05cc7570fd464e89f4"}
May 28 23:13:45.895721 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:45.895286    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" event={"ID":"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e","Type":"ContainerStarted","Data":"0a699c7aa5c04a48eab316feae6acd05f211a6e6a14f35a46f6286256a17c502"}
May 28 23:13:45.895721 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:45.895318    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" event={"ID":"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e","Type":"ContainerStarted","Data":"2847c9fc2b1e30c10028df4b83355f11a901a93555518b5374b0ac9ebd5553e3"}
May 28 23:13:45.895721 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:45.895331    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" event={"ID":"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e","Type":"ContainerStarted","Data":"00140154fe4d94d280d2e91da6c52875f351fd8ac9a6ec2d13fdbeb30dfddd85"}
May 28 23:13:45.980717 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:45.980677    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:45.980927 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:45.980731    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:45.980927 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:45.980746    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:45.980927 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:45.980761    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:45.983227 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:45.983071    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.32:8816: connect: connection refused"
May 28 23:13:45.983227 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:45.983119    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused"
May 28 23:13:45.983227 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:45.983118    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 28 23:13:45.983227 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:45.983121    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused"
May 28 23:13:48.981698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:48.981651    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused"
May 28 23:13:48.982308 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:48.981668    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.32:8816: connect: connection refused"
May 28 23:13:48.982428 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:48.981651    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused"
May 28 23:13:48.982428 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:48.981757    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 28 23:13:51.620798 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:51.620756    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused"
May 28 23:13:51.621266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:51.620743    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.31:8443: connect: connection refused"
May 28 23:13:51.981304 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:51.981222    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:51.981492 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:51.981466    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:51.981641 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:51.981626    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:51.982004 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:51.981981    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:51.982096 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:51.982015    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:51.982096 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:51.982029    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:51.982391 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:51.982370    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:51.982465 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:51.982416    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:51.982465 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:51.982445    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:51.986057 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:51.986036    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:52.007837 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:52.007780    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podStartSLOduration=19.007762917 podStartE2EDuration="19.007762917s" podCreationTimestamp="2026-05-28 23:13:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:13:45.922117806 +0000 UTC m=+699.115549058" watchObservedRunningTime="2026-05-28 23:13:52.007762917 +0000 UTC m=+705.201194168"
May 28 23:13:52.922122 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:52.922080    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:13:52.925139 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:13:52.925112    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:14:01.620857 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:01.620748    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.31:8443: connect: connection refused"
May 28 23:14:01.620857 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:01.620817    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused"
May 28 23:14:01.621327 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:01.620962    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:14:01.621327 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:01.621025    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:14:02.960005 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:02.959924    2577 generic.go:358] "Generic (PLEG): container finished" podID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerID="4b87909cc860623fd7062f5c9432bf6f653f165b0d79a0dbf72af86ad872cab1" exitCode=137
May 28 23:14:02.960005 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:02.959947    2577 generic.go:358] "Generic (PLEG): container finished" podID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerID="141b400a99ef6b11a5be2dff63a17946144852876d2cfed601f8b2e27ec56ad9" exitCode=137
May 28 23:14:02.960005 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:02.959993    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" event={"ID":"5da4d307-d559-49f3-b2e6-8c4da71e2c96","Type":"ContainerDied","Data":"4b87909cc860623fd7062f5c9432bf6f653f165b0d79a0dbf72af86ad872cab1"}
May 28 23:14:02.960401 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:02.960025    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" event={"ID":"5da4d307-d559-49f3-b2e6-8c4da71e2c96","Type":"ContainerDied","Data":"141b400a99ef6b11a5be2dff63a17946144852876d2cfed601f8b2e27ec56ad9"}
May 28 23:14:03.346998 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.346977    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:14:03.448145 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.448094    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-online-tls\") pod \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") "
May 28 23:14:03.448343 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.448167    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5da4d307-d559-49f3-b2e6-8c4da71e2c96-feast-data\") pod \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") "
May 28 23:14:03.448343 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.448195    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs9s8\" (UniqueName: \"kubernetes.io/projected/5da4d307-d559-49f3-b2e6-8c4da71e2c96-kube-api-access-fs9s8\") pod \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") "
May 28 23:14:03.448343 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.448329    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-offline-tls\") pod \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") "
May 28 23:14:03.448520 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.448371    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-ui-tls\") pod \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") "
May 28 23:14:03.448520 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.448423    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-registry-tls\") pod \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\" (UID: \"5da4d307-d559-49f3-b2e6-8c4da71e2c96\") "
May 28 23:14:03.448719 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.448687    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da4d307-d559-49f3-b2e6-8c4da71e2c96-feast-data" (OuterVolumeSpecName: "feast-data") pod "5da4d307-d559-49f3-b2e6-8c4da71e2c96" (UID: "5da4d307-d559-49f3-b2e6-8c4da71e2c96"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:14:03.450565 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.450539    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da4d307-d559-49f3-b2e6-8c4da71e2c96-kube-api-access-fs9s8" (OuterVolumeSpecName: "kube-api-access-fs9s8") pod "5da4d307-d559-49f3-b2e6-8c4da71e2c96" (UID: "5da4d307-d559-49f3-b2e6-8c4da71e2c96"). InnerVolumeSpecName "kube-api-access-fs9s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:14:03.450734 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.450708    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "5da4d307-d559-49f3-b2e6-8c4da71e2c96" (UID: "5da4d307-d559-49f3-b2e6-8c4da71e2c96"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:14:03.450791 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.450717    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5da4d307-d559-49f3-b2e6-8c4da71e2c96" (UID: "5da4d307-d559-49f3-b2e6-8c4da71e2c96"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:14:03.450925 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.450905    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5da4d307-d559-49f3-b2e6-8c4da71e2c96" (UID: "5da4d307-d559-49f3-b2e6-8c4da71e2c96"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:14:03.450987 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.450968    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-online-tls" (OuterVolumeSpecName: "online-tls") pod "5da4d307-d559-49f3-b2e6-8c4da71e2c96" (UID: "5da4d307-d559-49f3-b2e6-8c4da71e2c96"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:14:03.550085 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.550054    2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5da4d307-d559-49f3-b2e6-8c4da71e2c96-feast-data\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:14:03.550085 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.550085    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs9s8\" (UniqueName: \"kubernetes.io/projected/5da4d307-d559-49f3-b2e6-8c4da71e2c96-kube-api-access-fs9s8\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:14:03.550252 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.550099    2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-offline-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:14:03.550252 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.550108    2577 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-ui-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:14:03.550252 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.550116    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-registry-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:14:03.550252 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.550124    2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5da4d307-d559-49f3-b2e6-8c4da71e2c96-online-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:14:03.966830 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.966750    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"
May 28 23:14:03.967251 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.966750    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px" event={"ID":"5da4d307-d559-49f3-b2e6-8c4da71e2c96","Type":"ContainerDied","Data":"6b09d34f46da0f69a9aa13b04fd96b5f709b546f8d4d3bae3450750db4315b3e"}
May 28 23:14:03.967251 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.966889    2577 scope.go:117] "RemoveContainer" containerID="4565e0a62c3d4567d2fe36b142214523c616a136eb9e8985640b8dcc8b4238dc"
May 28 23:14:03.976044 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.976021    2577 scope.go:117] "RemoveContainer" containerID="4b87909cc860623fd7062f5c9432bf6f653f165b0d79a0dbf72af86ad872cab1"
May 28 23:14:03.983445 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.983428    2577 scope.go:117] "RemoveContainer" containerID="edcc991f03d8ba26a49bdf1f2e05705d0dcfd4696e9f77222a8f22a8c41530d1"
May 28 23:14:03.987899 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.987857    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"]
May 28 23:14:03.992158 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.992131    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f8c865756-kv9px"]
May 28 23:14:03.992923 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.992902    2577 scope.go:117] "RemoveContainer" containerID="141b400a99ef6b11a5be2dff63a17946144852876d2cfed601f8b2e27ec56ad9"
May 28 23:14:03.999990 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:03.999973    2577 scope.go:117] "RemoveContainer" containerID="c6504d6412efd70444cbab2bbe9ad72fe5a62e795c611a76c93c5faadec50a89"
May 28 23:14:04.007327 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:04.007314    2577 scope.go:117] "RemoveContainer" containerID="9ffe729a419929e59ddc180e2e51c94b35c5c162c913a924202afd3bacad14ac"
May 28 23:14:05.491884 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:05.491843    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" path="/var/lib/kubelet/pods/5da4d307-d559-49f3-b2e6-8c4da71e2c96/volumes"
May 28 23:14:54.620813 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.620772    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"]
May 28 23:14:54.621436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621243    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="offline"
May 28 23:14:54.621436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621260    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="offline"
May 28 23:14:54.621436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621275    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="online"
May 28 23:14:54.621436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621284    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="online"
May 28 23:14:54.621436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621299    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="feast-init"
May 28 23:14:54.621436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621308    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="feast-init"
May 28 23:14:54.621436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621333    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="feast-apply"
May 28 23:14:54.621436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621339    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="feast-apply"
May 28 23:14:54.621436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621349    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="registry"
May 28 23:14:54.621436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621355    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="registry"
May 28 23:14:54.621436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621367    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="ui"
May 28 23:14:54.621436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621377    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="ui"
May 28 23:14:54.621436 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621441    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="online"
May 28 23:14:54.622087 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621454    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="ui"
May 28 23:14:54.622087 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621466    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="registry"
May 28 23:14:54.622087 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.621476    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="5da4d307-d559-49f3-b2e6-8c4da71e2c96" containerName="offline"
May 28 23:14:54.625044 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.625026    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.627847 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.627824    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 28 23:14:54.628044 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.628000    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-7tm49\""
May 28 23:14:54.628044 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.628026    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 28 23:14:54.628170 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.628058    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\""
May 28 23:14:54.628170 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.627892    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 28 23:14:54.628170 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.627922    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 28 23:14:54.628170 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.627910    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 28 23:14:54.628329 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.627981    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 28 23:14:54.637833 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.637808    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"]
May 28 23:14:54.685832 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.685797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-offline-tls\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.685832 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.685832    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-online-tls\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.686063 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.685976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e0026d8f-878d-48e4-a39b-a55af48a0a10-feast-data\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.686063 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.686010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e0026d8f-878d-48e4-a39b-a55af48a0a10-registry-tls\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.686063 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.686031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45ctt\" (UniqueName: \"kubernetes.io/projected/e0026d8f-878d-48e4-a39b-a55af48a0a10-kube-api-access-45ctt\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.686220 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.686104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-ui-tls\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.787520 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.787489    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e0026d8f-878d-48e4-a39b-a55af48a0a10-feast-data\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.787520 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.787525    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e0026d8f-878d-48e4-a39b-a55af48a0a10-registry-tls\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.787765 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.787547    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-45ctt\" (UniqueName: \"kubernetes.io/projected/e0026d8f-878d-48e4-a39b-a55af48a0a10-kube-api-access-45ctt\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.787765 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.787576    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-ui-tls\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.787765 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.787600    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-offline-tls\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.787765 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.787615    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-online-tls\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.788018 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.787983    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e0026d8f-878d-48e4-a39b-a55af48a0a10-feast-data\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.788245 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.788226    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e0026d8f-878d-48e4-a39b-a55af48a0a10-registry-tls\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.790189 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.790161    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-ui-tls\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.790305 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.790269    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-online-tls\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.790305 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.790284    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-offline-tls\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.796497 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.796474    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45ctt\" (UniqueName: \"kubernetes.io/projected/e0026d8f-878d-48e4-a39b-a55af48a0a10-kube-api-access-45ctt\") pod \"feast-simple-feast-remote-setup-5d4bf9878d-56cn8\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:54.937077 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:54.936991    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:14:55.073347 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:55.073320    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"]
May 28 23:14:55.075158 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:14:55.075128    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0026d8f_878d_48e4_a39b_a55af48a0a10.slice/crio-7946428a95b061fb6067b0c0950242be8a1a06096c570e7841e4b422fdbc1141 WatchSource:0}: Error finding container 7946428a95b061fb6067b0c0950242be8a1a06096c570e7841e4b422fdbc1141: Status 404 returned error can't find the container with id 7946428a95b061fb6067b0c0950242be8a1a06096c570e7841e4b422fdbc1141
May 28 23:14:55.136317 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:55.136285    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" event={"ID":"e0026d8f-878d-48e4-a39b-a55af48a0a10","Type":"ContainerStarted","Data":"7946428a95b061fb6067b0c0950242be8a1a06096c570e7841e4b422fdbc1141"}
May 28 23:14:56.141914 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:14:56.141874    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" event={"ID":"e0026d8f-878d-48e4-a39b-a55af48a0a10","Type":"ContainerStarted","Data":"74fad5237835d1b2fd556ae511ad62c1fe2ad03106293e913e0b4621bc418cbd"}
May 28 23:15:00.155909 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:00.155844    2577 generic.go:358] "Generic (PLEG): container finished" podID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerID="74fad5237835d1b2fd556ae511ad62c1fe2ad03106293e913e0b4621bc418cbd" exitCode=0
May 28 23:15:00.156296 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:00.155920    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" event={"ID":"e0026d8f-878d-48e4-a39b-a55af48a0a10","Type":"ContainerDied","Data":"74fad5237835d1b2fd556ae511ad62c1fe2ad03106293e913e0b4621bc418cbd"}
May 28 23:15:01.161975 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:01.161930    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" event={"ID":"e0026d8f-878d-48e4-a39b-a55af48a0a10","Type":"ContainerStarted","Data":"b0ea501475af358f8b9d252453abe2bb85d0f31a7eda3cd173f792b55dc345c4"}
May 28 23:15:06.180666 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:06.180628    2577 generic.go:358] "Generic (PLEG): container finished" podID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerID="b0ea501475af358f8b9d252453abe2bb85d0f31a7eda3cd173f792b55dc345c4" exitCode=0
May 28 23:15:06.181071 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:06.180703    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" event={"ID":"e0026d8f-878d-48e4-a39b-a55af48a0a10","Type":"ContainerDied","Data":"b0ea501475af358f8b9d252453abe2bb85d0f31a7eda3cd173f792b55dc345c4"}
May 28 23:15:07.188425 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:07.188376    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" event={"ID":"e0026d8f-878d-48e4-a39b-a55af48a0a10","Type":"ContainerStarted","Data":"0fec2f941c077def6669cd3968f6ee99af1b661b2c773d7d1d78604a57c607d0"}
May 28 23:15:07.188937 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:07.188435    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" event={"ID":"e0026d8f-878d-48e4-a39b-a55af48a0a10","Type":"ContainerStarted","Data":"c44c98326a0e05b21ef393904988ca8a272cf6053cacdcfbbb70e3d4d33f1065"}
May 28 23:15:07.188937 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:07.188452    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" event={"ID":"e0026d8f-878d-48e4-a39b-a55af48a0a10","Type":"ContainerStarted","Data":"8958e6f8506ec9f76caf7a1917194ee17a18ca10a89b8b2c4c15baa224752557"}
May 28 23:15:07.214349 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:07.214287    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" podStartSLOduration=13.214267571 podStartE2EDuration="13.214267571s" podCreationTimestamp="2026-05-28 23:14:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:15:07.21255828 +0000 UTC m=+780.405989556" watchObservedRunningTime="2026-05-28 23:15:07.214267571 +0000 UTC m=+780.407698824"
May 28 23:15:09.938086 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:09.938041    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:15:09.938550 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:09.938216    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:15:09.938550 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:09.938237    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:15:09.940103 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:09.940068    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused"
May 28 23:15:09.940358 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:09.940105    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.33:8816: connect: connection refused"
May 28 23:15:09.940459 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:09.940120    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 28 23:15:12.938677 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:12.938644    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:15:12.939077 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:12.938758    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:15:12.939212 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:12.939190    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:15:12.939279 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:12.939225    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:15:12.939378 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:12.939364    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:15:12.939668 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:12.939652    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:15:12.942738 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:12.942723    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:15:13.212485 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:13.212404    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:15:13.215387 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:15:13.215365    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:16:15.896640 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:15.896601    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"]
May 28 23:16:15.897157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:15.896959    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="online" containerID="cri-o://8958e6f8506ec9f76caf7a1917194ee17a18ca10a89b8b2c4c15baa224752557" gracePeriod=30
May 28 23:16:15.897157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:15.896994    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="offline" containerID="cri-o://c44c98326a0e05b21ef393904988ca8a272cf6053cacdcfbbb70e3d4d33f1065" gracePeriod=30
May 28 23:16:15.897157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:15.897125    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="ui" containerID="cri-o://0fec2f941c077def6669cd3968f6ee99af1b661b2c773d7d1d78604a57c607d0" gracePeriod=30
May 28 23:16:16.010774 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:16.010737    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"]
May 28 23:16:16.424157 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:16.424121    2577 generic.go:358] "Generic (PLEG): container finished" podID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerID="0fec2f941c077def6669cd3968f6ee99af1b661b2c773d7d1d78604a57c607d0" exitCode=0
May 28 23:16:16.424327 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:16.424178    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" event={"ID":"e0026d8f-878d-48e4-a39b-a55af48a0a10","Type":"ContainerDied","Data":"0fec2f941c077def6669cd3968f6ee99af1b661b2c773d7d1d78604a57c607d0"}
May 28 23:16:16.424527 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:16.424499    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="registry" containerID="cri-o://00140154fe4d94d280d2e91da6c52875f351fd8ac9a6ec2d13fdbeb30dfddd85" gracePeriod=30
May 28 23:16:16.424666 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:16.424504    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="online" containerID="cri-o://2847c9fc2b1e30c10028df4b83355f11a901a93555518b5374b0ac9ebd5553e3" gracePeriod=30
May 28 23:16:16.424666 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:16.424505    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="offline" containerID="cri-o://0a699c7aa5c04a48eab316feae6acd05f211a6e6a14f35a46f6286256a17c502" gracePeriod=30
May 28 23:16:16.424796 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:16.424505    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="ui" containerID="cri-o://8a065089d0d76b7850aaab807cb4f8de9c56386560dbac05cc7570fd464e89f4" gracePeriod=30
May 28 23:16:17.430336 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:17.430305    2577 generic.go:358] "Generic (PLEG): container finished" podID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerID="8a065089d0d76b7850aaab807cb4f8de9c56386560dbac05cc7570fd464e89f4" exitCode=0
May 28 23:16:17.430336 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:17.430333    2577 generic.go:358] "Generic (PLEG): container finished" podID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerID="2847c9fc2b1e30c10028df4b83355f11a901a93555518b5374b0ac9ebd5553e3" exitCode=0
May 28 23:16:17.430797 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:17.430382    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" event={"ID":"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e","Type":"ContainerDied","Data":"8a065089d0d76b7850aaab807cb4f8de9c56386560dbac05cc7570fd464e89f4"}
May 28 23:16:17.430797 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:17.430413    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" event={"ID":"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e","Type":"ContainerDied","Data":"2847c9fc2b1e30c10028df4b83355f11a901a93555518b5374b0ac9ebd5553e3"}
May 28 23:16:17.432568 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:17.432542    2577 generic.go:358] "Generic (PLEG): container finished" podID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerID="8958e6f8506ec9f76caf7a1917194ee17a18ca10a89b8b2c4c15baa224752557" exitCode=0
May 28 23:16:17.432676 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:17.432575    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" event={"ID":"e0026d8f-878d-48e4-a39b-a55af48a0a10","Type":"ContainerDied","Data":"8958e6f8506ec9f76caf7a1917194ee17a18ca10a89b8b2c4c15baa224752557"}
May 28 23:16:21.982688 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:21.982640    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused"
May 28 23:16:22.923131 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:22.923089    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 28 23:16:22.939306 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:22.939273    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused"
May 28 23:16:23.213271 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:23.213183    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 28 23:16:31.982772 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:31.982722    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused"
May 28 23:16:32.922617 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:32.922570    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 28 23:16:32.939370 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:32.939328    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused"
May 28 23:16:33.212834 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:33.212738    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 28 23:16:41.982375 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:41.982327    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused"
May 28 23:16:41.982826 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:41.982463    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:16:42.923259 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:42.923216    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 28 23:16:42.923428 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:42.923340    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:16:42.939753 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:42.939715    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused"
May 28 23:16:42.939918 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:42.939817    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:16:43.212966 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:43.212840    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 28 23:16:43.213338 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:43.213047    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:16:46.085855 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.085831    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:16:46.164151 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.164059    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-offline-tls\") pod \"e0026d8f-878d-48e4-a39b-a55af48a0a10\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") "
May 28 23:16:46.164151 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.164112    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e0026d8f-878d-48e4-a39b-a55af48a0a10-feast-data\") pod \"e0026d8f-878d-48e4-a39b-a55af48a0a10\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") "
May 28 23:16:46.164359 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.164177    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-online-tls\") pod \"e0026d8f-878d-48e4-a39b-a55af48a0a10\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") "
May 28 23:16:46.164359 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.164246    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e0026d8f-878d-48e4-a39b-a55af48a0a10-registry-tls\") pod \"e0026d8f-878d-48e4-a39b-a55af48a0a10\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") "
May 28 23:16:46.164359 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.164319    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45ctt\" (UniqueName: \"kubernetes.io/projected/e0026d8f-878d-48e4-a39b-a55af48a0a10-kube-api-access-45ctt\") pod \"e0026d8f-878d-48e4-a39b-a55af48a0a10\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") "
May 28 23:16:46.164359 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.164352    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-ui-tls\") pod \"e0026d8f-878d-48e4-a39b-a55af48a0a10\" (UID: \"e0026d8f-878d-48e4-a39b-a55af48a0a10\") "
May 28 23:16:46.164673 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.164636    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0026d8f-878d-48e4-a39b-a55af48a0a10-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e0026d8f-878d-48e4-a39b-a55af48a0a10" (UID: "e0026d8f-878d-48e4-a39b-a55af48a0a10"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:16:46.164756 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.164738    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0026d8f-878d-48e4-a39b-a55af48a0a10-feast-data" (OuterVolumeSpecName: "feast-data") pod "e0026d8f-878d-48e4-a39b-a55af48a0a10" (UID: "e0026d8f-878d-48e4-a39b-a55af48a0a10"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:16:46.166538 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.166486    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e0026d8f-878d-48e4-a39b-a55af48a0a10" (UID: "e0026d8f-878d-48e4-a39b-a55af48a0a10"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:16:46.166671 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.166566    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-online-tls" (OuterVolumeSpecName: "online-tls") pod "e0026d8f-878d-48e4-a39b-a55af48a0a10" (UID: "e0026d8f-878d-48e4-a39b-a55af48a0a10"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:16:46.166671 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.166579    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0026d8f-878d-48e4-a39b-a55af48a0a10-kube-api-access-45ctt" (OuterVolumeSpecName: "kube-api-access-45ctt") pod "e0026d8f-878d-48e4-a39b-a55af48a0a10" (UID: "e0026d8f-878d-48e4-a39b-a55af48a0a10"). InnerVolumeSpecName "kube-api-access-45ctt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:16:46.166671 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.166620    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e0026d8f-878d-48e4-a39b-a55af48a0a10" (UID: "e0026d8f-878d-48e4-a39b-a55af48a0a10"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:16:46.265475 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.265443    2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e0026d8f-878d-48e4-a39b-a55af48a0a10-feast-data\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:16:46.265475 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.265468    2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-online-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:16:46.265475 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.265479    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/e0026d8f-878d-48e4-a39b-a55af48a0a10-registry-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:16:46.265723 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.265487    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45ctt\" (UniqueName: \"kubernetes.io/projected/e0026d8f-878d-48e4-a39b-a55af48a0a10-kube-api-access-45ctt\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:16:46.265723 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.265497    2577 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-ui-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:16:46.265723 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.265506    2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e0026d8f-878d-48e4-a39b-a55af48a0a10-offline-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:16:46.528292 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.528264    2577 generic.go:358] "Generic (PLEG): container finished" podID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerID="0a699c7aa5c04a48eab316feae6acd05f211a6e6a14f35a46f6286256a17c502" exitCode=137
May 28 23:16:46.528292 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.528288    2577 generic.go:358] "Generic (PLEG): container finished" podID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerID="00140154fe4d94d280d2e91da6c52875f351fd8ac9a6ec2d13fdbeb30dfddd85" exitCode=137
May 28 23:16:46.528492 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.528340    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" event={"ID":"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e","Type":"ContainerDied","Data":"0a699c7aa5c04a48eab316feae6acd05f211a6e6a14f35a46f6286256a17c502"}
May 28 23:16:46.528492 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.528373    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" event={"ID":"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e","Type":"ContainerDied","Data":"00140154fe4d94d280d2e91da6c52875f351fd8ac9a6ec2d13fdbeb30dfddd85"}
May 28 23:16:46.530421 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.530399    2577 generic.go:358] "Generic (PLEG): container finished" podID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerID="c44c98326a0e05b21ef393904988ca8a272cf6053cacdcfbbb70e3d4d33f1065" exitCode=137
May 28 23:16:46.530528 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.530432    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" event={"ID":"e0026d8f-878d-48e4-a39b-a55af48a0a10","Type":"ContainerDied","Data":"c44c98326a0e05b21ef393904988ca8a272cf6053cacdcfbbb70e3d4d33f1065"}
May 28 23:16:46.530528 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.530470    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"
May 28 23:16:46.530528 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.530476    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8" event={"ID":"e0026d8f-878d-48e4-a39b-a55af48a0a10","Type":"ContainerDied","Data":"7946428a95b061fb6067b0c0950242be8a1a06096c570e7841e4b422fdbc1141"}
May 28 23:16:46.530528 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.530501    2577 scope.go:117] "RemoveContainer" containerID="0fec2f941c077def6669cd3968f6ee99af1b661b2c773d7d1d78604a57c607d0"
May 28 23:16:46.540614 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.540595    2577 scope.go:117] "RemoveContainer" containerID="c44c98326a0e05b21ef393904988ca8a272cf6053cacdcfbbb70e3d4d33f1065"
May 28 23:16:46.548032 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.548017    2577 scope.go:117] "RemoveContainer" containerID="8958e6f8506ec9f76caf7a1917194ee17a18ca10a89b8b2c4c15baa224752557"
May 28 23:16:46.554740 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.554711    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"]
May 28 23:16:46.556717 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.556682    2577 scope.go:117] "RemoveContainer" containerID="b0ea501475af358f8b9d252453abe2bb85d0f31a7eda3cd173f792b55dc345c4"
May 28 23:16:46.558466 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.558444    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5d4bf9878d-56cn8"]
May 28 23:16:46.563975 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.563941    2577 scope.go:117] "RemoveContainer" containerID="74fad5237835d1b2fd556ae511ad62c1fe2ad03106293e913e0b4621bc418cbd"
May 28 23:16:46.570787 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.570770    2577 scope.go:117] "RemoveContainer" containerID="0fec2f941c077def6669cd3968f6ee99af1b661b2c773d7d1d78604a57c607d0"
May 28 23:16:46.571061 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:16:46.571041    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fec2f941c077def6669cd3968f6ee99af1b661b2c773d7d1d78604a57c607d0\": container with ID starting with 0fec2f941c077def6669cd3968f6ee99af1b661b2c773d7d1d78604a57c607d0 not found: ID does not exist" containerID="0fec2f941c077def6669cd3968f6ee99af1b661b2c773d7d1d78604a57c607d0"
May 28 23:16:46.571116 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.571068    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fec2f941c077def6669cd3968f6ee99af1b661b2c773d7d1d78604a57c607d0"} err="failed to get container status \"0fec2f941c077def6669cd3968f6ee99af1b661b2c773d7d1d78604a57c607d0\": rpc error: code = NotFound desc = could not find container \"0fec2f941c077def6669cd3968f6ee99af1b661b2c773d7d1d78604a57c607d0\": container with ID starting with 0fec2f941c077def6669cd3968f6ee99af1b661b2c773d7d1d78604a57c607d0 not found: ID does not exist"
May 28 23:16:46.571116 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.571085    2577 scope.go:117] "RemoveContainer" containerID="c44c98326a0e05b21ef393904988ca8a272cf6053cacdcfbbb70e3d4d33f1065"
May 28 23:16:46.571306 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:16:46.571290    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44c98326a0e05b21ef393904988ca8a272cf6053cacdcfbbb70e3d4d33f1065\": container with ID starting with c44c98326a0e05b21ef393904988ca8a272cf6053cacdcfbbb70e3d4d33f1065 not found: ID does not exist" containerID="c44c98326a0e05b21ef393904988ca8a272cf6053cacdcfbbb70e3d4d33f1065"
May 28 23:16:46.571349 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.571311    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44c98326a0e05b21ef393904988ca8a272cf6053cacdcfbbb70e3d4d33f1065"} err="failed to get container status \"c44c98326a0e05b21ef393904988ca8a272cf6053cacdcfbbb70e3d4d33f1065\": rpc error: code = NotFound desc = could not find container \"c44c98326a0e05b21ef393904988ca8a272cf6053cacdcfbbb70e3d4d33f1065\": container with ID starting with c44c98326a0e05b21ef393904988ca8a272cf6053cacdcfbbb70e3d4d33f1065 not found: ID does not exist"
May 28 23:16:46.571349 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.571325    2577 scope.go:117] "RemoveContainer" containerID="8958e6f8506ec9f76caf7a1917194ee17a18ca10a89b8b2c4c15baa224752557"
May 28 23:16:46.571527 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:16:46.571508    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8958e6f8506ec9f76caf7a1917194ee17a18ca10a89b8b2c4c15baa224752557\": container with ID starting with 8958e6f8506ec9f76caf7a1917194ee17a18ca10a89b8b2c4c15baa224752557 not found: ID does not exist" containerID="8958e6f8506ec9f76caf7a1917194ee17a18ca10a89b8b2c4c15baa224752557"
May 28 23:16:46.571579 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.571532    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8958e6f8506ec9f76caf7a1917194ee17a18ca10a89b8b2c4c15baa224752557"} err="failed to get container status \"8958e6f8506ec9f76caf7a1917194ee17a18ca10a89b8b2c4c15baa224752557\": rpc error: code = NotFound desc = could not find container \"8958e6f8506ec9f76caf7a1917194ee17a18ca10a89b8b2c4c15baa224752557\": container with ID starting with 8958e6f8506ec9f76caf7a1917194ee17a18ca10a89b8b2c4c15baa224752557 not found: ID does not exist"
May 28 23:16:46.571579 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.571548    2577 scope.go:117] "RemoveContainer" containerID="b0ea501475af358f8b9d252453abe2bb85d0f31a7eda3cd173f792b55dc345c4"
May 28 23:16:46.571742 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:16:46.571725    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ea501475af358f8b9d252453abe2bb85d0f31a7eda3cd173f792b55dc345c4\": container with ID starting with b0ea501475af358f8b9d252453abe2bb85d0f31a7eda3cd173f792b55dc345c4 not found: ID does not exist" containerID="b0ea501475af358f8b9d252453abe2bb85d0f31a7eda3cd173f792b55dc345c4"
May 28 23:16:46.571780 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.571745    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ea501475af358f8b9d252453abe2bb85d0f31a7eda3cd173f792b55dc345c4"} err="failed to get container status \"b0ea501475af358f8b9d252453abe2bb85d0f31a7eda3cd173f792b55dc345c4\": rpc error: code = NotFound desc = could not find container \"b0ea501475af358f8b9d252453abe2bb85d0f31a7eda3cd173f792b55dc345c4\": container with ID starting with b0ea501475af358f8b9d252453abe2bb85d0f31a7eda3cd173f792b55dc345c4 not found: ID does not exist"
May 28 23:16:46.571780 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.571757    2577 scope.go:117] "RemoveContainer" containerID="74fad5237835d1b2fd556ae511ad62c1fe2ad03106293e913e0b4621bc418cbd"
May 28 23:16:46.571960 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:16:46.571943    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74fad5237835d1b2fd556ae511ad62c1fe2ad03106293e913e0b4621bc418cbd\": container with ID starting with 74fad5237835d1b2fd556ae511ad62c1fe2ad03106293e913e0b4621bc418cbd not found: ID does not exist" containerID="74fad5237835d1b2fd556ae511ad62c1fe2ad03106293e913e0b4621bc418cbd"
May 28 23:16:46.572013 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:46.571960    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74fad5237835d1b2fd556ae511ad62c1fe2ad03106293e913e0b4621bc418cbd"} err="failed to get container status \"74fad5237835d1b2fd556ae511ad62c1fe2ad03106293e913e0b4621bc418cbd\": rpc error: code = NotFound desc = could not find container \"74fad5237835d1b2fd556ae511ad62c1fe2ad03106293e913e0b4621bc418cbd\": container with ID starting with 74fad5237835d1b2fd556ae511ad62c1fe2ad03106293e913e0b4621bc418cbd not found: ID does not exist"
May 28 23:16:47.084440 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.084419    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:16:47.175072 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.174995    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-registry-tls\") pod \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") "
May 28 23:16:47.175453 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.175075    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxvm7\" (UniqueName: \"kubernetes.io/projected/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-kube-api-access-bxvm7\") pod \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") "
May 28 23:16:47.175453 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.175179    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-online-tls\") pod \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") "
May 28 23:16:47.175453 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.175219    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-feast-data\") pod \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") "
May 28 23:16:47.175453 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.175245    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-ui-tls\") pod \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") "
May 28 23:16:47.175453 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.175275    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-offline-tls\") pod \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\" (UID: \"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e\") "
May 28 23:16:47.175727 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.175695    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-feast-data" (OuterVolumeSpecName: "feast-data") pod "08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" (UID: "08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:16:47.177492 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.177459    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" (UID: "08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:16:47.177492 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.177484    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" (UID: "08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:16:47.177643 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.177505    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-online-tls" (OuterVolumeSpecName: "online-tls") pod "08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" (UID: "08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:16:47.177729 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.177711    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" (UID: "08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:16:47.177779 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.177760    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-kube-api-access-bxvm7" (OuterVolumeSpecName: "kube-api-access-bxvm7") pod "08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" (UID: "08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e"). InnerVolumeSpecName "kube-api-access-bxvm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:16:47.276221 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.276187    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxvm7\" (UniqueName: \"kubernetes.io/projected/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-kube-api-access-bxvm7\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:16:47.276221 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.276210    2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-online-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:16:47.276221 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.276220    2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-feast-data\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:16:47.276221 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.276227    2577 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-ui-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:16:47.276504 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.276235    2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-offline-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:16:47.276504 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.276242    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e-registry-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:16:47.491267 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.491240    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" path="/var/lib/kubelet/pods/e0026d8f-878d-48e4-a39b-a55af48a0a10/volumes"
May 28 23:16:47.537407 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.537372    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf" event={"ID":"08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e","Type":"ContainerDied","Data":"026248f1d6eaf8c6636a2e7e10212770a0e09d7f6f5f9d0c5fdd34b75bd6a936"}
May 28 23:16:47.537407 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.537413    2577 scope.go:117] "RemoveContainer" containerID="8a065089d0d76b7850aaab807cb4f8de9c56386560dbac05cc7570fd464e89f4"
May 28 23:16:47.537611 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.537454    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"
May 28 23:16:47.545954 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.545929    2577 scope.go:117] "RemoveContainer" containerID="0a699c7aa5c04a48eab316feae6acd05f211a6e6a14f35a46f6286256a17c502"
May 28 23:16:47.553105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.553082    2577 scope.go:117] "RemoveContainer" containerID="2847c9fc2b1e30c10028df4b83355f11a901a93555518b5374b0ac9ebd5553e3"
May 28 23:16:47.557521 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.557491    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"]
May 28 23:16:47.561600 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.561580    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f8c865756-ftlbf"]
May 28 23:16:47.563650 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.563638    2577 scope.go:117] "RemoveContainer" containerID="00140154fe4d94d280d2e91da6c52875f351fd8ac9a6ec2d13fdbeb30dfddd85"
May 28 23:16:47.571845 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.571829    2577 scope.go:117] "RemoveContainer" containerID="7a00017724a4b1813c37304bd7d47da8a7245ffa0f795e48dfb00f4a0eff823b"
May 28 23:16:47.578726 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:47.578707    2577 scope.go:117] "RemoveContainer" containerID="23e3d1ef4e8408603617a2482eeed5af37a5b9b74038684e60d50c4209a1a104"
May 28 23:16:49.491879 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:49.491841    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" path="/var/lib/kubelet/pods/08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e/volumes"
May 28 23:16:58.785054 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.784976    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-c9rv7"]
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785330    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="ui"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785346    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="ui"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785362    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="registry"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785372    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="registry"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785383    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="feast-apply"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785391    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="feast-apply"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785405    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="offline"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785413    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="offline"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785429    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="feast-init"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785438    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="feast-init"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785454    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="online"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785462    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="online"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785473    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="offline"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785481    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="offline"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785491    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="feast-apply"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785499    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="feast-apply"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785512    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="online"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785519    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="online"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785532    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="feast-init"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785540    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="feast-init"
May 28 23:16:58.785542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785550    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="ui"
May 28 23:16:58.786306 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785559    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="ui"
May 28 23:16:58.786306 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785645    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="online"
May 28 23:16:58.786306 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785659    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="offline"
May 28 23:16:58.786306 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785668    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="offline"
May 28 23:16:58.786306 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785678    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="online"
May 28 23:16:58.786306 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785691    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="registry"
May 28 23:16:58.786306 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785704    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="08b2f6b1-a7f1-49a7-a64d-3235b4be0d1e" containerName="ui"
May 28 23:16:58.786306 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.785713    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="e0026d8f-878d-48e4-a39b-a55af48a0a10" containerName="ui"
May 28 23:16:58.788692 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.788668    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-c9rv7"
May 28 23:16:58.791096 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.791072    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 23:16:58.791201 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.791108    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\""
May 28 23:16:58.792084 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.792064    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-lk24v\""
May 28 23:16:58.792192 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.792072    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 23:16:58.795824 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.795258    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-c9rv7"]
May 28 23:16:58.876453 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.876426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/3706e68f-6c3b-44bb-81a7-d6a937762d7f-postgresdata\") pod \"postgres-7c7656cc6d-c9rv7\" (UID: \"3706e68f-6c3b-44bb-81a7-d6a937762d7f\") " pod="test-ns-feast/postgres-7c7656cc6d-c9rv7"
May 28 23:16:58.876546 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.876480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzb6w\" (UniqueName: \"kubernetes.io/projected/3706e68f-6c3b-44bb-81a7-d6a937762d7f-kube-api-access-wzb6w\") pod \"postgres-7c7656cc6d-c9rv7\" (UID: \"3706e68f-6c3b-44bb-81a7-d6a937762d7f\") " pod="test-ns-feast/postgres-7c7656cc6d-c9rv7"
May 28 23:16:58.977020 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.976968    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/3706e68f-6c3b-44bb-81a7-d6a937762d7f-postgresdata\") pod \"postgres-7c7656cc6d-c9rv7\" (UID: \"3706e68f-6c3b-44bb-81a7-d6a937762d7f\") " pod="test-ns-feast/postgres-7c7656cc6d-c9rv7"
May 28 23:16:58.977191 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.977046    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wzb6w\" (UniqueName: \"kubernetes.io/projected/3706e68f-6c3b-44bb-81a7-d6a937762d7f-kube-api-access-wzb6w\") pod \"postgres-7c7656cc6d-c9rv7\" (UID: \"3706e68f-6c3b-44bb-81a7-d6a937762d7f\") " pod="test-ns-feast/postgres-7c7656cc6d-c9rv7"
May 28 23:16:58.977368 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.977346    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/3706e68f-6c3b-44bb-81a7-d6a937762d7f-postgresdata\") pod \"postgres-7c7656cc6d-c9rv7\" (UID: \"3706e68f-6c3b-44bb-81a7-d6a937762d7f\") " pod="test-ns-feast/postgres-7c7656cc6d-c9rv7"
May 28 23:16:58.986057 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:58.986037    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzb6w\" (UniqueName: \"kubernetes.io/projected/3706e68f-6c3b-44bb-81a7-d6a937762d7f-kube-api-access-wzb6w\") pod \"postgres-7c7656cc6d-c9rv7\" (UID: \"3706e68f-6c3b-44bb-81a7-d6a937762d7f\") " pod="test-ns-feast/postgres-7c7656cc6d-c9rv7"
May 28 23:16:59.126644 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:59.126550    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-c9rv7"
May 28 23:16:59.255495 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:59.255432    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-c9rv7"]
May 28 23:16:59.258415 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:16:59.258371    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3706e68f_6c3b_44bb_81a7_d6a937762d7f.slice/crio-2be46224afddf2d08ea81ad2756c2f1980c6609a8824c3715ddec170d4d6205c WatchSource:0}: Error finding container 2be46224afddf2d08ea81ad2756c2f1980c6609a8824c3715ddec170d4d6205c: Status 404 returned error can't find the container with id 2be46224afddf2d08ea81ad2756c2f1980c6609a8824c3715ddec170d4d6205c
May 28 23:16:59.581525 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:16:59.581492    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-c9rv7" event={"ID":"3706e68f-6c3b-44bb-81a7-d6a937762d7f","Type":"ContainerStarted","Data":"2be46224afddf2d08ea81ad2756c2f1980c6609a8824c3715ddec170d4d6205c"}
May 28 23:17:06.607162 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:06.607118    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-c9rv7" event={"ID":"3706e68f-6c3b-44bb-81a7-d6a937762d7f","Type":"ContainerStarted","Data":"0d5c40daa28243dec719951eb5dafac3ba7ccbb4cdb284797e4c42a8fbb65029"}
May 28 23:17:06.621699 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:06.621646    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/postgres-7c7656cc6d-c9rv7" podStartSLOduration=2.332193707 podStartE2EDuration="8.621613506s" podCreationTimestamp="2026-05-28 23:16:58 +0000 UTC" firstStartedPulling="2026-05-28 23:16:59.260205739 +0000 UTC m=+892.453636969" lastFinishedPulling="2026-05-28 23:17:05.549625535 +0000 UTC m=+898.743056768" observedRunningTime="2026-05-28 23:17:06.621535353 +0000 UTC m=+899.814966607" watchObservedRunningTime="2026-05-28 23:17:06.621613506 +0000 UTC m=+899.815044761"
May 28 23:17:09.517467 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.517425    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"]
May 28 23:17:09.524163 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.524134    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.526938 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.526900    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\""
May 28 23:17:09.527055 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.526900    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\""
May 28 23:17:09.527650 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.527634    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 28 23:17:09.527724 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.527637    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\""
May 28 23:17:09.527724 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.527637    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-s95cc\""
May 28 23:17:09.532528 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.532505    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"]
May 28 23:17:09.572227 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.572194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-online-tls\") pod \"feast-credit-scoring-fdbdb4595-px7kf\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") " pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.572373 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.572235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-offline-tls\") pod \"feast-credit-scoring-fdbdb4595-px7kf\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") " pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.572373 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.572304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qltmz\" (UniqueName: \"kubernetes.io/projected/c4276bca-89bf-4971-a34c-40b2b743ab11-kube-api-access-qltmz\") pod \"feast-credit-scoring-fdbdb4595-px7kf\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") " pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.572373 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.572335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-registry-tls\") pod \"feast-credit-scoring-fdbdb4595-px7kf\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") " pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.572478 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.572398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c4276bca-89bf-4971-a34c-40b2b743ab11-feast-data\") pod \"feast-credit-scoring-fdbdb4595-px7kf\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") " pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.673200 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.673162    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-online-tls\") pod \"feast-credit-scoring-fdbdb4595-px7kf\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") " pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.673200 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.673203    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-offline-tls\") pod \"feast-credit-scoring-fdbdb4595-px7kf\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") " pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.673378 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.673234    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qltmz\" (UniqueName: \"kubernetes.io/projected/c4276bca-89bf-4971-a34c-40b2b743ab11-kube-api-access-qltmz\") pod \"feast-credit-scoring-fdbdb4595-px7kf\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") " pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.673378 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.673263    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-registry-tls\") pod \"feast-credit-scoring-fdbdb4595-px7kf\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") " pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.673378 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.673293    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c4276bca-89bf-4971-a34c-40b2b743ab11-feast-data\") pod \"feast-credit-scoring-fdbdb4595-px7kf\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") " pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.673762 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.673727    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c4276bca-89bf-4971-a34c-40b2b743ab11-feast-data\") pod \"feast-credit-scoring-fdbdb4595-px7kf\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") " pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.675905 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.675859    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-offline-tls\") pod \"feast-credit-scoring-fdbdb4595-px7kf\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") " pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.675988 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.675931    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-online-tls\") pod \"feast-credit-scoring-fdbdb4595-px7kf\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") " pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.675988 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.675942    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-registry-tls\") pod \"feast-credit-scoring-fdbdb4595-px7kf\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") " pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.682193 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.682165    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qltmz\" (UniqueName: \"kubernetes.io/projected/c4276bca-89bf-4971-a34c-40b2b743ab11-kube-api-access-qltmz\") pod \"feast-credit-scoring-fdbdb4595-px7kf\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") " pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.836341 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.836261    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:09.971159 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:09.971131    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"]
May 28 23:17:09.973889 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:17:09.973818    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4276bca_89bf_4971_a34c_40b2b743ab11.slice/crio-9190cb9ba9d62a339c4f6b4afe0004a972f9ceffd319e5bf8842dc2ca2f8d8fb WatchSource:0}: Error finding container 9190cb9ba9d62a339c4f6b4afe0004a972f9ceffd319e5bf8842dc2ca2f8d8fb: Status 404 returned error can't find the container with id 9190cb9ba9d62a339c4f6b4afe0004a972f9ceffd319e5bf8842dc2ca2f8d8fb
May 28 23:17:10.622538 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:10.622504    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" event={"ID":"c4276bca-89bf-4971-a34c-40b2b743ab11","Type":"ContainerStarted","Data":"4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393"}
May 28 23:17:10.622538 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:10.622540    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" event={"ID":"c4276bca-89bf-4971-a34c-40b2b743ab11","Type":"ContainerStarted","Data":"9190cb9ba9d62a339c4f6b4afe0004a972f9ceffd319e5bf8842dc2ca2f8d8fb"}
May 28 23:17:11.626949 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:11.626909    2577 generic.go:358] "Generic (PLEG): container finished" podID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerID="4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393" exitCode=0
May 28 23:17:11.627424 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:11.626990    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" event={"ID":"c4276bca-89bf-4971-a34c-40b2b743ab11","Type":"ContainerDied","Data":"4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393"}
May 28 23:17:12.634580 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:12.634536    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" event={"ID":"c4276bca-89bf-4971-a34c-40b2b743ab11","Type":"ContainerStarted","Data":"0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31"}
May 28 23:17:12.998661 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:12.990617    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:17:12.998661 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:12.994373    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:17:18.654280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:18.654246    2577 generic.go:358] "Generic (PLEG): container finished" podID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerID="0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31" exitCode=0
May 28 23:17:18.654776 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:18.654325    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" event={"ID":"c4276bca-89bf-4971-a34c-40b2b743ab11","Type":"ContainerDied","Data":"0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31"}
May 28 23:17:18.655559 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:18.655544    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 23:17:19.661468 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:19.661426    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" event={"ID":"c4276bca-89bf-4971-a34c-40b2b743ab11","Type":"ContainerStarted","Data":"ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147"}
May 28 23:17:19.661988 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:19.661478    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" event={"ID":"c4276bca-89bf-4971-a34c-40b2b743ab11","Type":"ContainerStarted","Data":"1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e"}
May 28 23:17:19.661988 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:19.661497    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" event={"ID":"c4276bca-89bf-4971-a34c-40b2b743ab11","Type":"ContainerStarted","Data":"60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a"}
May 28 23:17:19.687782 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:19.687715    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" podStartSLOduration=10.687694744 podStartE2EDuration="10.687694744s" podCreationTimestamp="2026-05-28 23:17:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:17:19.686307779 +0000 UTC m=+912.879739033" watchObservedRunningTime="2026-05-28 23:17:19.687694744 +0000 UTC m=+912.881125998"
May 28 23:17:19.837107 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:19.837069    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:19.837270 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:19.837127    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:19.837270 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:19.837142    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:21.837077 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:21.837040    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:21.837546 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:21.837157    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:21.837546 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:21.837176    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:21.839048 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:21.839007    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.35:6571: connect: connection refused"
May 28 23:17:21.839191 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:21.839091    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 28 23:17:21.839191 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:21.839007    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.35:8816: connect: connection refused"
May 28 23:17:24.837036 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:24.836989    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.35:6571: connect: connection refused"
May 28 23:17:24.838051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:24.838023    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:24.838962 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:24.838938    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:24.842554 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:24.842533    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:25.686373 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:25.686347    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:27.837379 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:27.837335    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.35:6571: connect: connection refused"
May 28 23:17:30.838067 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:30.838033    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:17:30.838986 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:17:30.838960    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:18:31.211031 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.210952    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"]
May 28 23:18:31.211508 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.211292    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="registry" containerID="cri-o://60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a" gracePeriod=30
May 28 23:18:31.211508 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.211352    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="online" containerID="cri-o://1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e" gracePeriod=30
May 28 23:18:31.211508 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.211355    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="offline" containerID="cri-o://ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147" gracePeriod=30
May 28 23:18:31.227564 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.227538    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-c9rv7"]
May 28 23:18:31.227734 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.227716    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/postgres-7c7656cc6d-c9rv7" podUID="3706e68f-6c3b-44bb-81a7-d6a937762d7f" containerName="postgres" containerID="cri-o://0d5c40daa28243dec719951eb5dafac3ba7ccbb4cdb284797e4c42a8fbb65029" gracePeriod=30
May 28 23:18:31.476009 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.475983    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-c9rv7"
May 28 23:18:31.597540 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.597503    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/3706e68f-6c3b-44bb-81a7-d6a937762d7f-postgresdata\") pod \"3706e68f-6c3b-44bb-81a7-d6a937762d7f\" (UID: \"3706e68f-6c3b-44bb-81a7-d6a937762d7f\") "
May 28 23:18:31.597707 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.597586    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzb6w\" (UniqueName: \"kubernetes.io/projected/3706e68f-6c3b-44bb-81a7-d6a937762d7f-kube-api-access-wzb6w\") pod \"3706e68f-6c3b-44bb-81a7-d6a937762d7f\" (UID: \"3706e68f-6c3b-44bb-81a7-d6a937762d7f\") "
May 28 23:18:31.599921 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.599859    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3706e68f-6c3b-44bb-81a7-d6a937762d7f-kube-api-access-wzb6w" (OuterVolumeSpecName: "kube-api-access-wzb6w") pod "3706e68f-6c3b-44bb-81a7-d6a937762d7f" (UID: "3706e68f-6c3b-44bb-81a7-d6a937762d7f"). InnerVolumeSpecName "kube-api-access-wzb6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:18:31.626849 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.626803    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3706e68f-6c3b-44bb-81a7-d6a937762d7f-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "3706e68f-6c3b-44bb-81a7-d6a937762d7f" (UID: "3706e68f-6c3b-44bb-81a7-d6a937762d7f"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:18:31.698991 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.698947    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzb6w\" (UniqueName: \"kubernetes.io/projected/3706e68f-6c3b-44bb-81a7-d6a937762d7f-kube-api-access-wzb6w\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:18:31.698991 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.698984    2577 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/3706e68f-6c3b-44bb-81a7-d6a937762d7f-postgresdata\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:18:31.906095 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.906067    2577 generic.go:358] "Generic (PLEG): container finished" podID="3706e68f-6c3b-44bb-81a7-d6a937762d7f" containerID="0d5c40daa28243dec719951eb5dafac3ba7ccbb4cdb284797e4c42a8fbb65029" exitCode=0
May 28 23:18:31.906220 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.906125    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-c9rv7"
May 28 23:18:31.906220 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.906149    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-c9rv7" event={"ID":"3706e68f-6c3b-44bb-81a7-d6a937762d7f","Type":"ContainerDied","Data":"0d5c40daa28243dec719951eb5dafac3ba7ccbb4cdb284797e4c42a8fbb65029"}
May 28 23:18:31.906220 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.906201    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-c9rv7" event={"ID":"3706e68f-6c3b-44bb-81a7-d6a937762d7f","Type":"ContainerDied","Data":"2be46224afddf2d08ea81ad2756c2f1980c6609a8824c3715ddec170d4d6205c"}
May 28 23:18:31.906369 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.906220    2577 scope.go:117] "RemoveContainer" containerID="0d5c40daa28243dec719951eb5dafac3ba7ccbb4cdb284797e4c42a8fbb65029"
May 28 23:18:31.922583 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.922563    2577 scope.go:117] "RemoveContainer" containerID="0d5c40daa28243dec719951eb5dafac3ba7ccbb4cdb284797e4c42a8fbb65029"
May 28 23:18:31.922856 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:18:31.922834    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5c40daa28243dec719951eb5dafac3ba7ccbb4cdb284797e4c42a8fbb65029\": container with ID starting with 0d5c40daa28243dec719951eb5dafac3ba7ccbb4cdb284797e4c42a8fbb65029 not found: ID does not exist" containerID="0d5c40daa28243dec719951eb5dafac3ba7ccbb4cdb284797e4c42a8fbb65029"
May 28 23:18:31.922960 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.922878    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5c40daa28243dec719951eb5dafac3ba7ccbb4cdb284797e4c42a8fbb65029"} err="failed to get container status \"0d5c40daa28243dec719951eb5dafac3ba7ccbb4cdb284797e4c42a8fbb65029\": rpc error: code = NotFound desc = could not find container \"0d5c40daa28243dec719951eb5dafac3ba7ccbb4cdb284797e4c42a8fbb65029\": container with ID starting with 0d5c40daa28243dec719951eb5dafac3ba7ccbb4cdb284797e4c42a8fbb65029 not found: ID does not exist"
May 28 23:18:31.926020 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.925998    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-c9rv7"]
May 28 23:18:31.931601 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:31.931579    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-c9rv7"]
May 28 23:18:32.912898 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:32.912850    2577 generic.go:358] "Generic (PLEG): container finished" podID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerID="1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e" exitCode=0
May 28 23:18:32.913259 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:32.912924    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" event={"ID":"c4276bca-89bf-4971-a34c-40b2b743ab11","Type":"ContainerDied","Data":"1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e"}
May 28 23:18:33.492001 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:33.491969    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3706e68f-6c3b-44bb-81a7-d6a937762d7f" path="/var/lib/kubelet/pods/3706e68f-6c3b-44bb-81a7-d6a937762d7f/volumes"
May 28 23:18:35.683691 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:35.683647    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 28 23:18:45.683605 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:45.683558    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 28 23:18:55.684254 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:55.684210    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 28 23:18:55.684699 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:18:55.684350    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:19:01.864553 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:01.864528    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:19:01.956564 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:01.956526    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c4276bca-89bf-4971-a34c-40b2b743ab11-feast-data\") pod \"c4276bca-89bf-4971-a34c-40b2b743ab11\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") "
May 28 23:19:01.956753 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:01.956580    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qltmz\" (UniqueName: \"kubernetes.io/projected/c4276bca-89bf-4971-a34c-40b2b743ab11-kube-api-access-qltmz\") pod \"c4276bca-89bf-4971-a34c-40b2b743ab11\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") "
May 28 23:19:01.956753 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:01.956640    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-online-tls\") pod \"c4276bca-89bf-4971-a34c-40b2b743ab11\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") "
May 28 23:19:01.956753 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:01.956659    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-registry-tls\") pod \"c4276bca-89bf-4971-a34c-40b2b743ab11\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") "
May 28 23:19:01.956753 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:01.956695    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-offline-tls\") pod \"c4276bca-89bf-4971-a34c-40b2b743ab11\" (UID: \"c4276bca-89bf-4971-a34c-40b2b743ab11\") "
May 28 23:19:01.958909 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:01.958858    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "c4276bca-89bf-4971-a34c-40b2b743ab11" (UID: "c4276bca-89bf-4971-a34c-40b2b743ab11"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:19:01.959003 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:01.958903    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c4276bca-89bf-4971-a34c-40b2b743ab11" (UID: "c4276bca-89bf-4971-a34c-40b2b743ab11"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:19:01.959311 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:01.959292    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4276bca-89bf-4971-a34c-40b2b743ab11-kube-api-access-qltmz" (OuterVolumeSpecName: "kube-api-access-qltmz") pod "c4276bca-89bf-4971-a34c-40b2b743ab11" (UID: "c4276bca-89bf-4971-a34c-40b2b743ab11"). InnerVolumeSpecName "kube-api-access-qltmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:19:01.959367 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:01.959302    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-online-tls" (OuterVolumeSpecName: "online-tls") pod "c4276bca-89bf-4971-a34c-40b2b743ab11" (UID: "c4276bca-89bf-4971-a34c-40b2b743ab11"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:19:01.964775 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:01.964751    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4276bca-89bf-4971-a34c-40b2b743ab11-feast-data" (OuterVolumeSpecName: "feast-data") pod "c4276bca-89bf-4971-a34c-40b2b743ab11" (UID: "c4276bca-89bf-4971-a34c-40b2b743ab11"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:19:02.012273 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.012240    2577 generic.go:358] "Generic (PLEG): container finished" podID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerID="ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147" exitCode=137
May 28 23:19:02.012273 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.012270    2577 generic.go:358] "Generic (PLEG): container finished" podID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerID="60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a" exitCode=137
May 28 23:19:02.012486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.012322    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"
May 28 23:19:02.012486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.012322    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" event={"ID":"c4276bca-89bf-4971-a34c-40b2b743ab11","Type":"ContainerDied","Data":"ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147"}
May 28 23:19:02.012486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.012364    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" event={"ID":"c4276bca-89bf-4971-a34c-40b2b743ab11","Type":"ContainerDied","Data":"60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a"}
May 28 23:19:02.012486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.012376    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf" event={"ID":"c4276bca-89bf-4971-a34c-40b2b743ab11","Type":"ContainerDied","Data":"9190cb9ba9d62a339c4f6b4afe0004a972f9ceffd319e5bf8842dc2ca2f8d8fb"}
May 28 23:19:02.012486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.012390    2577 scope.go:117] "RemoveContainer" containerID="ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147"
May 28 23:19:02.022841 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.022817    2577 scope.go:117] "RemoveContainer" containerID="1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e"
May 28 23:19:02.031068 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.031051    2577 scope.go:117] "RemoveContainer" containerID="60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a"
May 28 23:19:02.040028 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.039744    2577 scope.go:117] "RemoveContainer" containerID="0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31"
May 28 23:19:02.040028 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.040022    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"]
May 28 23:19:02.042214 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.042189    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-fdbdb4595-px7kf"]
May 28 23:19:02.047540 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.047519    2577 scope.go:117] "RemoveContainer" containerID="4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393"
May 28 23:19:02.054741 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.054723    2577 scope.go:117] "RemoveContainer" containerID="ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147"
May 28 23:19:02.055053 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:19:02.055030    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147\": container with ID starting with ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147 not found: ID does not exist" containerID="ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147"
May 28 23:19:02.055105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.055069    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147"} err="failed to get container status \"ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147\": rpc error: code = NotFound desc = could not find container \"ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147\": container with ID starting with ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147 not found: ID does not exist"
May 28 23:19:02.055105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.055089    2577 scope.go:117] "RemoveContainer" containerID="1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e"
May 28 23:19:02.055330 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:19:02.055301    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e\": container with ID starting with 1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e not found: ID does not exist" containerID="1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e"
May 28 23:19:02.055405 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.055335    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e"} err="failed to get container status \"1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e\": rpc error: code = NotFound desc = could not find container \"1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e\": container with ID starting with 1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e not found: ID does not exist"
May 28 23:19:02.055405 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.055357    2577 scope.go:117] "RemoveContainer" containerID="60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a"
May 28 23:19:02.055606 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:19:02.055589    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a\": container with ID starting with 60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a not found: ID does not exist" containerID="60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a"
May 28 23:19:02.055669 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.055613    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a"} err="failed to get container status \"60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a\": rpc error: code = NotFound desc = could not find container \"60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a\": container with ID starting with 60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a not found: ID does not exist"
May 28 23:19:02.055669 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.055633    2577 scope.go:117] "RemoveContainer" containerID="0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31"
May 28 23:19:02.055888 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:19:02.055848    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31\": container with ID starting with 0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31 not found: ID does not exist" containerID="0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31"
May 28 23:19:02.055949 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.055889    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31"} err="failed to get container status \"0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31\": rpc error: code = NotFound desc = could not find container \"0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31\": container with ID starting with 0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31 not found: ID does not exist"
May 28 23:19:02.055949 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.055904    2577 scope.go:117] "RemoveContainer" containerID="4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393"
May 28 23:19:02.056133 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:19:02.056116    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393\": container with ID starting with 4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393 not found: ID does not exist" containerID="4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393"
May 28 23:19:02.056176 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.056137    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393"} err="failed to get container status \"4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393\": rpc error: code = NotFound desc = could not find container \"4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393\": container with ID starting with 4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393 not found: ID does not exist"
May 28 23:19:02.056176 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.056161    2577 scope.go:117] "RemoveContainer" containerID="ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147"
May 28 23:19:02.056382 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.056364    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147"} err="failed to get container status \"ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147\": rpc error: code = NotFound desc = could not find container \"ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147\": container with ID starting with ce986198d28f5f96353953bcd38af235973742aa0d1f8b494a0701e2078e7147 not found: ID does not exist"
May 28 23:19:02.056417 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.056384    2577 scope.go:117] "RemoveContainer" containerID="1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e"
May 28 23:19:02.056595 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.056578    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e"} err="failed to get container status \"1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e\": rpc error: code = NotFound desc = could not find container \"1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e\": container with ID starting with 1d17e9348966bad87549cea7a69655f86ebd3f63f85b42f05aa270190108e44e not found: ID does not exist"
May 28 23:19:02.056633 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.056596    2577 scope.go:117] "RemoveContainer" containerID="60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a"
May 28 23:19:02.056808 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.056789    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a"} err="failed to get container status \"60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a\": rpc error: code = NotFound desc = could not find container \"60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a\": container with ID starting with 60e97ac2fa5a0a6be73ccba0b8ade4d680632620e700fe3bb013ce2af3156c0a not found: ID does not exist"
May 28 23:19:02.056847 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.056809    2577 scope.go:117] "RemoveContainer" containerID="0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31"
May 28 23:19:02.057028 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.057011    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31"} err="failed to get container status \"0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31\": rpc error: code = NotFound desc = could not find container \"0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31\": container with ID starting with 0af54496888b0683d411339032a5d6d222035833fef493481f541ed28a67ef31 not found: ID does not exist"
May 28 23:19:02.057082 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.057030    2577 scope.go:117] "RemoveContainer" containerID="4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393"
May 28 23:19:02.057210 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.057191    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393"} err="failed to get container status \"4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393\": rpc error: code = NotFound desc = could not find container \"4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393\": container with ID starting with 4c92fb6414b2382b4474b591760b97ff5f3ad782016dc344564ff3ddff880393 not found: ID does not exist"
May 28 23:19:02.057291 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.057277    2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-offline-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:19:02.057329 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.057295    2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c4276bca-89bf-4971-a34c-40b2b743ab11-feast-data\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:19:02.057329 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.057304    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qltmz\" (UniqueName: \"kubernetes.io/projected/c4276bca-89bf-4971-a34c-40b2b743ab11-kube-api-access-qltmz\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:19:02.057329 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.057313    2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-online-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:19:02.057329 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:02.057321    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c4276bca-89bf-4971-a34c-40b2b743ab11-registry-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:19:03.491981 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:19:03.491940    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" path="/var/lib/kubelet/pods/c4276bca-89bf-4971-a34c-40b2b743ab11/volumes"
May 28 23:20:55.829794 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.829760    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-854544c579-kjdds"]
May 28 23:20:55.830280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830136    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="online"
May 28 23:20:55.830280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830149    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="online"
May 28 23:20:55.830280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830159    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="feast-init"
May 28 23:20:55.830280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830164    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="feast-init"
May 28 23:20:55.830280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830172    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3706e68f-6c3b-44bb-81a7-d6a937762d7f" containerName="postgres"
May 28 23:20:55.830280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830177    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3706e68f-6c3b-44bb-81a7-d6a937762d7f" containerName="postgres"
May 28 23:20:55.830280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830188    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="offline"
May 28 23:20:55.830280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830196    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="offline"
May 28 23:20:55.830280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830216    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="registry"
May 28 23:20:55.830280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830220    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="registry"
May 28 23:20:55.830280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830228    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="feast-apply"
May 28 23:20:55.830280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830233    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="feast-apply"
May 28 23:20:55.830280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830289    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="registry"
May 28 23:20:55.830855 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830300    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="3706e68f-6c3b-44bb-81a7-d6a937762d7f" containerName="postgres"
May 28 23:20:55.830855 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830306    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="offline"
May 28 23:20:55.830855 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.830312    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="c4276bca-89bf-4971-a34c-40b2b743ab11" containerName="online"
May 28 23:20:55.833344 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.833320    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-kjdds"
May 28 23:20:55.836451 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.836428    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\""
May 28 23:20:55.836574 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.836460    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\""
May 28 23:20:55.837232 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.837210    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-j85qn\""
May 28 23:20:55.846606 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.846581    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-kjdds"]
May 28 23:20:55.876153 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.876127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xz7m\" (UniqueName: \"kubernetes.io/projected/eb76630e-9170-4ccb-ac46-2c7adca56e4a-kube-api-access-2xz7m\") pod \"redis-854544c579-kjdds\" (UID: \"eb76630e-9170-4ccb-ac46-2c7adca56e4a\") " pod="test-ns-feast-rest/redis-854544c579-kjdds"
May 28 23:20:55.976950 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.976911    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2xz7m\" (UniqueName: \"kubernetes.io/projected/eb76630e-9170-4ccb-ac46-2c7adca56e4a-kube-api-access-2xz7m\") pod \"redis-854544c579-kjdds\" (UID: \"eb76630e-9170-4ccb-ac46-2c7adca56e4a\") " pod="test-ns-feast-rest/redis-854544c579-kjdds"
May 28 23:20:55.986141 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:55.986114    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xz7m\" (UniqueName: \"kubernetes.io/projected/eb76630e-9170-4ccb-ac46-2c7adca56e4a-kube-api-access-2xz7m\") pod \"redis-854544c579-kjdds\" (UID: \"eb76630e-9170-4ccb-ac46-2c7adca56e4a\") " pod="test-ns-feast-rest/redis-854544c579-kjdds"
May 28 23:20:56.153468 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:56.153380    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-kjdds"
May 28 23:20:56.282502 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:56.282472    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-kjdds"]
May 28 23:20:56.285182 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:20:56.285143    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb76630e_9170_4ccb_ac46_2c7adca56e4a.slice/crio-7eccc48edffdc31f42fa373847edd2356e9aa9461b211b00c17bfd910c14ca14 WatchSource:0}: Error finding container 7eccc48edffdc31f42fa373847edd2356e9aa9461b211b00c17bfd910c14ca14: Status 404 returned error can't find the container with id 7eccc48edffdc31f42fa373847edd2356e9aa9461b211b00c17bfd910c14ca14
May 28 23:20:56.391957 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:56.391918    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-kjdds" event={"ID":"eb76630e-9170-4ccb-ac46-2c7adca56e4a","Type":"ContainerStarted","Data":"7eccc48edffdc31f42fa373847edd2356e9aa9461b211b00c17bfd910c14ca14"}
May 28 23:20:58.400561 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:58.400527    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-kjdds" event={"ID":"eb76630e-9170-4ccb-ac46-2c7adca56e4a","Type":"ContainerStarted","Data":"0fa73033209011385ac51945fc9687a1b1192b9ac7f4051563147f1a2afab1d1"}
May 28 23:20:58.415513 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:20:58.415468    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/redis-854544c579-kjdds" podStartSLOduration=2.104362272 podStartE2EDuration="3.415451866s" podCreationTimestamp="2026-05-28 23:20:55 +0000 UTC" firstStartedPulling="2026-05-28 23:20:56.286842307 +0000 UTC m=+1129.480273537" lastFinishedPulling="2026-05-28 23:20:57.597931895 +0000 UTC m=+1130.791363131" observedRunningTime="2026-05-28 23:20:58.41458242 +0000 UTC m=+1131.608013674" watchObservedRunningTime="2026-05-28 23:20:58.415451866 +0000 UTC m=+1131.608883117"
May 28 23:21:01.283432 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:01.283400    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"]
May 28 23:21:01.286613 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:01.286594    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"
May 28 23:21:01.289024 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:01.289001    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\""
May 28 23:21:01.297207 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:01.297179    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"]
May 28 23:21:01.317422 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:01.317391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4trwq\" (UniqueName: \"kubernetes.io/projected/0be901ac-1cf0-45d2-98a8-fad87021c676-kube-api-access-4trwq\") pod \"postgres-7c7656cc6d-lrhg2\" (UID: \"0be901ac-1cf0-45d2-98a8-fad87021c676\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"
May 28 23:21:01.317562 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:01.317481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/0be901ac-1cf0-45d2-98a8-fad87021c676-postgresdata\") pod \"postgres-7c7656cc6d-lrhg2\" (UID: \"0be901ac-1cf0-45d2-98a8-fad87021c676\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"
May 28 23:21:01.418423 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:01.418389    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4trwq\" (UniqueName: \"kubernetes.io/projected/0be901ac-1cf0-45d2-98a8-fad87021c676-kube-api-access-4trwq\") pod \"postgres-7c7656cc6d-lrhg2\" (UID: \"0be901ac-1cf0-45d2-98a8-fad87021c676\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"
May 28 23:21:01.418604 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:01.418469    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/0be901ac-1cf0-45d2-98a8-fad87021c676-postgresdata\") pod \"postgres-7c7656cc6d-lrhg2\" (UID: \"0be901ac-1cf0-45d2-98a8-fad87021c676\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"
May 28 23:21:01.418845 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:01.418829    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/0be901ac-1cf0-45d2-98a8-fad87021c676-postgresdata\") pod \"postgres-7c7656cc6d-lrhg2\" (UID: \"0be901ac-1cf0-45d2-98a8-fad87021c676\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"
May 28 23:21:01.427219 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:01.427198    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4trwq\" (UniqueName: \"kubernetes.io/projected/0be901ac-1cf0-45d2-98a8-fad87021c676-kube-api-access-4trwq\") pod \"postgres-7c7656cc6d-lrhg2\" (UID: \"0be901ac-1cf0-45d2-98a8-fad87021c676\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"
May 28 23:21:01.596516 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:01.596431    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"
May 28 23:21:01.725113 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:01.725073    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"]
May 28 23:21:01.727820 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:21:01.727792    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0be901ac_1cf0_45d2_98a8_fad87021c676.slice/crio-e652cf43caebd1aed59a7c31153fadb9b3263e474b4b07fb11a2fbdfe628362b WatchSource:0}: Error finding container e652cf43caebd1aed59a7c31153fadb9b3263e474b4b07fb11a2fbdfe628362b: Status 404 returned error can't find the container with id e652cf43caebd1aed59a7c31153fadb9b3263e474b4b07fb11a2fbdfe628362b
May 28 23:21:02.414878 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:02.414833    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2" event={"ID":"0be901ac-1cf0-45d2-98a8-fad87021c676","Type":"ContainerStarted","Data":"00c4f897cf07bbd11bab966045847106920c888cb5c13a4d4e8e3432ce70933e"}
May 28 23:21:02.415245 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:02.414898    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2" event={"ID":"0be901ac-1cf0-45d2-98a8-fad87021c676","Type":"ContainerStarted","Data":"e652cf43caebd1aed59a7c31153fadb9b3263e474b4b07fb11a2fbdfe628362b"}
May 28 23:21:02.431268 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:02.431217    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2" podStartSLOduration=1.431202842 podStartE2EDuration="1.431202842s" podCreationTimestamp="2026-05-28 23:21:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:21:02.429548924 +0000 UTC m=+1135.622980181" watchObservedRunningTime="2026-05-28 23:21:02.431202842 +0000 UTC m=+1135.624634094"
May 28 23:21:06.941045 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:06.941006    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"]
May 28 23:21:06.946898 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:06.946855    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:06.949703 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:06.949678    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\""
May 28 23:21:06.949835 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:06.949679    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\""
May 28 23:21:06.950472 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:06.950449    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\""
May 28 23:21:06.950594 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:06.950449    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-p9q5n\""
May 28 23:21:06.956626 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:06.956597    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"]
May 28 23:21:06.961970 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:06.961915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3c40243b-8f1a-496d-8082-7bd7be78391c-registry-tls\") pod \"feast-credit-scoring-65b7c7ff87-c5bc2\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") " pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:06.962098 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:06.961983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3c40243b-8f1a-496d-8082-7bd7be78391c-online-tls\") pod \"feast-credit-scoring-65b7c7ff87-c5bc2\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") " pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:06.962098 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:06.962030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3c40243b-8f1a-496d-8082-7bd7be78391c-feast-data\") pod \"feast-credit-scoring-65b7c7ff87-c5bc2\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") " pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:06.962329 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:06.962305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xg9p\" (UniqueName: \"kubernetes.io/projected/3c40243b-8f1a-496d-8082-7bd7be78391c-kube-api-access-9xg9p\") pod \"feast-credit-scoring-65b7c7ff87-c5bc2\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") " pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:07.063398 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:07.063361    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9xg9p\" (UniqueName: \"kubernetes.io/projected/3c40243b-8f1a-496d-8082-7bd7be78391c-kube-api-access-9xg9p\") pod \"feast-credit-scoring-65b7c7ff87-c5bc2\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") " pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:07.063595 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:07.063436    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3c40243b-8f1a-496d-8082-7bd7be78391c-registry-tls\") pod \"feast-credit-scoring-65b7c7ff87-c5bc2\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") " pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:07.063595 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:07.063455    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3c40243b-8f1a-496d-8082-7bd7be78391c-online-tls\") pod \"feast-credit-scoring-65b7c7ff87-c5bc2\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") " pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:07.063595 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:07.063475    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3c40243b-8f1a-496d-8082-7bd7be78391c-feast-data\") pod \"feast-credit-scoring-65b7c7ff87-c5bc2\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") " pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:07.063822 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:07.063804    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3c40243b-8f1a-496d-8082-7bd7be78391c-feast-data\") pod \"feast-credit-scoring-65b7c7ff87-c5bc2\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") " pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:07.066029 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:07.066001    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3c40243b-8f1a-496d-8082-7bd7be78391c-registry-tls\") pod \"feast-credit-scoring-65b7c7ff87-c5bc2\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") " pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:07.066190 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:07.066167    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3c40243b-8f1a-496d-8082-7bd7be78391c-online-tls\") pod \"feast-credit-scoring-65b7c7ff87-c5bc2\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") " pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:07.072213 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:07.072183    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xg9p\" (UniqueName: \"kubernetes.io/projected/3c40243b-8f1a-496d-8082-7bd7be78391c-kube-api-access-9xg9p\") pod \"feast-credit-scoring-65b7c7ff87-c5bc2\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") " pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:07.262012 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:07.261968    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:07.420806 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:07.420778    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"]
May 28 23:21:07.423448 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:21:07.423416    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c40243b_8f1a_496d_8082_7bd7be78391c.slice/crio-1540392982f1d31fef6de0804e2ce995338ec21415ede29ef53b36bd56e18d93 WatchSource:0}: Error finding container 1540392982f1d31fef6de0804e2ce995338ec21415ede29ef53b36bd56e18d93: Status 404 returned error can't find the container with id 1540392982f1d31fef6de0804e2ce995338ec21415ede29ef53b36bd56e18d93
May 28 23:21:07.432682 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:07.432652    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" event={"ID":"3c40243b-8f1a-496d-8082-7bd7be78391c","Type":"ContainerStarted","Data":"1540392982f1d31fef6de0804e2ce995338ec21415ede29ef53b36bd56e18d93"}
May 28 23:21:08.438172 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:08.438133    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" event={"ID":"3c40243b-8f1a-496d-8082-7bd7be78391c","Type":"ContainerStarted","Data":"7cc7741ad7fcd7c2c607423c13198257bcb916ad282ec9c10b111da0b44f95b6"}
May 28 23:21:09.446566 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:09.446516    2577 generic.go:358] "Generic (PLEG): container finished" podID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerID="7cc7741ad7fcd7c2c607423c13198257bcb916ad282ec9c10b111da0b44f95b6" exitCode=0
May 28 23:21:09.446984 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:09.446604    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" event={"ID":"3c40243b-8f1a-496d-8082-7bd7be78391c","Type":"ContainerDied","Data":"7cc7741ad7fcd7c2c607423c13198257bcb916ad282ec9c10b111da0b44f95b6"}
May 28 23:21:10.451394 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:10.451356    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" event={"ID":"3c40243b-8f1a-496d-8082-7bd7be78391c","Type":"ContainerStarted","Data":"ec1e4336477ecd7973fa2147e818655ac97ba6a69a3bbc2e1c9c57f75cf698ef"}
May 28 23:21:17.476832 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:17.476794    2577 generic.go:358] "Generic (PLEG): container finished" podID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerID="ec1e4336477ecd7973fa2147e818655ac97ba6a69a3bbc2e1c9c57f75cf698ef" exitCode=0
May 28 23:21:17.477233 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:17.476892    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" event={"ID":"3c40243b-8f1a-496d-8082-7bd7be78391c","Type":"ContainerDied","Data":"ec1e4336477ecd7973fa2147e818655ac97ba6a69a3bbc2e1c9c57f75cf698ef"}
May 28 23:21:17.481451 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:17.481432    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\""
May 28 23:21:18.482828 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:18.482789    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" event={"ID":"3c40243b-8f1a-496d-8082-7bd7be78391c","Type":"ContainerStarted","Data":"d4fcba655967655d1fbabe39d42cc75f071c7deddfbd864789033ea5896864e5"}
May 28 23:21:18.483322 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:18.482836    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" event={"ID":"3c40243b-8f1a-496d-8082-7bd7be78391c","Type":"ContainerStarted","Data":"264b7e76f1f092b16ca112ac248e36a5ca31903b3eff13ac9e85392f940608f9"}
May 28 23:21:18.505350 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:18.505301    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" podStartSLOduration=12.505280522 podStartE2EDuration="12.505280522s" podCreationTimestamp="2026-05-28 23:21:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:21:18.50421607 +0000 UTC m=+1151.697647323" watchObservedRunningTime="2026-05-28 23:21:18.505280522 +0000 UTC m=+1151.698711776"
May 28 23:21:19.262491 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:19.262446    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:19.262491 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:19.262501    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:19.264118 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:19.264070    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.38:6571: connect: connection refused"
May 28 23:21:19.264242 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:19.264109    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused"
May 28 23:21:22.262947 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:22.262900    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.38:6571: connect: connection refused"
May 28 23:21:22.263376 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:22.263014    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused"
May 28 23:21:25.262499 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:25.262383    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.38:6571: connect: connection refused"
May 28 23:21:25.269718 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:25.269687    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:25.270115 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:25.270094    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:25.273792 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:25.273767    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:27.262175 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:27.262132    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:28.263285 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:28.263233    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.38:6571: connect: connection refused"
May 28 23:21:31.263556 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:31.263521    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:31.264343 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:31.264325    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:21:32.843930 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:32.843895    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"]
May 28 23:21:32.847706 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:32.847683    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:32.850336 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:32.850281    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-bzltx\""
May 28 23:21:32.850336 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:32.850281    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\""
May 28 23:21:32.851072 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:32.851055    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\""
May 28 23:21:32.851153 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:32.851097    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\""
May 28 23:21:32.860452 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:32.860430    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"]
May 28 23:21:33.002105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.002075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89091c32-5e43-4c1c-8042-d5750a52b714-online-tls\") pod \"feast-driver-ranking-5d85d8b884-xfn6l\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") " pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:33.002105 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.002110    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2wck\" (UniqueName: \"kubernetes.io/projected/89091c32-5e43-4c1c-8042-d5750a52b714-kube-api-access-s2wck\") pod \"feast-driver-ranking-5d85d8b884-xfn6l\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") " pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:33.002320 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.002162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/89091c32-5e43-4c1c-8042-d5750a52b714-registry-tls\") pod \"feast-driver-ranking-5d85d8b884-xfn6l\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") " pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:33.002320 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.002177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/89091c32-5e43-4c1c-8042-d5750a52b714-feast-data\") pod \"feast-driver-ranking-5d85d8b884-xfn6l\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") " pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:33.102719 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.102621    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89091c32-5e43-4c1c-8042-d5750a52b714-online-tls\") pod \"feast-driver-ranking-5d85d8b884-xfn6l\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") " pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:33.102719 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.102667    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s2wck\" (UniqueName: \"kubernetes.io/projected/89091c32-5e43-4c1c-8042-d5750a52b714-kube-api-access-s2wck\") pod \"feast-driver-ranking-5d85d8b884-xfn6l\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") " pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:33.102719 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.102700    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/89091c32-5e43-4c1c-8042-d5750a52b714-registry-tls\") pod \"feast-driver-ranking-5d85d8b884-xfn6l\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") " pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:33.102719 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.102715    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/89091c32-5e43-4c1c-8042-d5750a52b714-feast-data\") pod \"feast-driver-ranking-5d85d8b884-xfn6l\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") " pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:33.103139 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.103121    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/89091c32-5e43-4c1c-8042-d5750a52b714-feast-data\") pod \"feast-driver-ranking-5d85d8b884-xfn6l\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") " pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:33.105427 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.105406    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89091c32-5e43-4c1c-8042-d5750a52b714-online-tls\") pod \"feast-driver-ranking-5d85d8b884-xfn6l\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") " pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:33.105488 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.105424    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/89091c32-5e43-4c1c-8042-d5750a52b714-registry-tls\") pod \"feast-driver-ranking-5d85d8b884-xfn6l\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") " pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:33.111960 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.111930    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2wck\" (UniqueName: \"kubernetes.io/projected/89091c32-5e43-4c1c-8042-d5750a52b714-kube-api-access-s2wck\") pod \"feast-driver-ranking-5d85d8b884-xfn6l\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") " pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:33.157686 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.157651    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:33.299463 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.299437    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"]
May 28 23:21:33.301999 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:21:33.301969    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89091c32_5e43_4c1c_8042_d5750a52b714.slice/crio-21e99333fff7de879a61ff02a2e4fb5fb018ff0630e65844b26a56de5fb5ef15 WatchSource:0}: Error finding container 21e99333fff7de879a61ff02a2e4fb5fb018ff0630e65844b26a56de5fb5ef15: Status 404 returned error can't find the container with id 21e99333fff7de879a61ff02a2e4fb5fb018ff0630e65844b26a56de5fb5ef15
May 28 23:21:33.538355 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.538314    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" event={"ID":"89091c32-5e43-4c1c-8042-d5750a52b714","Type":"ContainerStarted","Data":"b8b3ea8a53307db50e01fb1edbe98adff715db26f03e6437f51d48e4a75470a9"}
May 28 23:21:33.538355 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:33.538359    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" event={"ID":"89091c32-5e43-4c1c-8042-d5750a52b714","Type":"ContainerStarted","Data":"21e99333fff7de879a61ff02a2e4fb5fb018ff0630e65844b26a56de5fb5ef15"}
May 28 23:21:38.557539 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:38.557501    2577 generic.go:358] "Generic (PLEG): container finished" podID="89091c32-5e43-4c1c-8042-d5750a52b714" containerID="b8b3ea8a53307db50e01fb1edbe98adff715db26f03e6437f51d48e4a75470a9" exitCode=0
May 28 23:21:38.557929 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:38.557572    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" event={"ID":"89091c32-5e43-4c1c-8042-d5750a52b714","Type":"ContainerDied","Data":"b8b3ea8a53307db50e01fb1edbe98adff715db26f03e6437f51d48e4a75470a9"}
May 28 23:21:39.562197 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:39.562161    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" event={"ID":"89091c32-5e43-4c1c-8042-d5750a52b714","Type":"ContainerStarted","Data":"e0fd4e98d1c00a50e89453f744ba52b71efc3a81d66073cbc58da7f35ed120a7"}
May 28 23:21:45.584793 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:45.584758    2577 generic.go:358] "Generic (PLEG): container finished" podID="89091c32-5e43-4c1c-8042-d5750a52b714" containerID="e0fd4e98d1c00a50e89453f744ba52b71efc3a81d66073cbc58da7f35ed120a7" exitCode=0
May 28 23:21:45.585304 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:45.584833    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" event={"ID":"89091c32-5e43-4c1c-8042-d5750a52b714","Type":"ContainerDied","Data":"e0fd4e98d1c00a50e89453f744ba52b71efc3a81d66073cbc58da7f35ed120a7"}
May 28 23:21:46.598311 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:46.598277    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" event={"ID":"89091c32-5e43-4c1c-8042-d5750a52b714","Type":"ContainerStarted","Data":"6fa069b1b9ba73b0d653ca5ff032097c81e77dfc4bcb0ad0e40932f888ccdd74"}
May 28 23:21:46.598311 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:46.598319    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" event={"ID":"89091c32-5e43-4c1c-8042-d5750a52b714","Type":"ContainerStarted","Data":"89dd3741e8781c359345d82a6d876466fb4df1800d642677c138d8bda6d37299"}
May 28 23:21:46.623745 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:46.623650    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" podStartSLOduration=14.623631149 podStartE2EDuration="14.623631149s" podCreationTimestamp="2026-05-28 23:21:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:21:46.622058038 +0000 UTC m=+1179.815489291" watchObservedRunningTime="2026-05-28 23:21:46.623631149 +0000 UTC m=+1179.817062404"
May 28 23:21:48.158274 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:48.158226    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:48.158274 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:48.158282    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:48.159769 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:48.159737    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.39:6571: connect: connection refused"
May 28 23:21:48.159931 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:48.159799    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused"
May 28 23:21:51.158569 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:51.158514    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.39:6571: connect: connection refused"
May 28 23:21:51.743564 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:51.743533    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:51.744007 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:51.743984    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:51.746943 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:51.746919    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:53.158423 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:53.158386    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:54.158590 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:54.158532    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.39:6571: connect: connection refused"
May 28 23:21:57.158841 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:57.158809    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:21:57.159629 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:21:57.159606    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:22:13.018845 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:22:13.018809    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:22:13.024226 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:22:13.024192    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:23:11.479209 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.479127    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs"]
May 28 23:23:11.479671 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.479443    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs" podUID="55638b90-c9f1-4397-986a-35ef1fa93654" containerName="manager" containerID="cri-o://0335c6314bf28f311ad7f9faf0ef791ab9a601cfedc88e8cfd7cdab6cd2c93b4" gracePeriod=10
May 28 23:23:11.696957 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.696925    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"]
May 28 23:23:11.697463 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.697284    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="registry" containerID="cri-o://264b7e76f1f092b16ca112ac248e36a5ca31903b3eff13ac9e85392f940608f9" gracePeriod=30
May 28 23:23:11.697463 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.697386    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="online" containerID="cri-o://d4fcba655967655d1fbabe39d42cc75f071c7deddfbd864789033ea5896864e5" gracePeriod=30
May 28 23:23:11.707560 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.707527    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"]
May 28 23:23:11.708351 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.708295    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="registry" containerID="cri-o://89dd3741e8781c359345d82a6d876466fb4df1800d642677c138d8bda6d37299" gracePeriod=30
May 28 23:23:11.708995 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.708973    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="online" containerID="cri-o://6fa069b1b9ba73b0d653ca5ff032097c81e77dfc4bcb0ad0e40932f888ccdd74" gracePeriod=30
May 28 23:23:11.767363 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.767333    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs"
May 28 23:23:11.876079 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.876047    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfhn2\" (UniqueName: \"kubernetes.io/projected/55638b90-c9f1-4397-986a-35ef1fa93654-kube-api-access-qfhn2\") pod \"55638b90-c9f1-4397-986a-35ef1fa93654\" (UID: \"55638b90-c9f1-4397-986a-35ef1fa93654\") "
May 28 23:23:11.878460 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.878432    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55638b90-c9f1-4397-986a-35ef1fa93654-kube-api-access-qfhn2" (OuterVolumeSpecName: "kube-api-access-qfhn2") pod "55638b90-c9f1-4397-986a-35ef1fa93654" (UID: "55638b90-c9f1-4397-986a-35ef1fa93654"). InnerVolumeSpecName "kube-api-access-qfhn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:23:11.898786 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.898744    2577 generic.go:358] "Generic (PLEG): container finished" podID="55638b90-c9f1-4397-986a-35ef1fa93654" containerID="0335c6314bf28f311ad7f9faf0ef791ab9a601cfedc88e8cfd7cdab6cd2c93b4" exitCode=0
May 28 23:23:11.898973 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.898820    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs"
May 28 23:23:11.898973 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.898825    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs" event={"ID":"55638b90-c9f1-4397-986a-35ef1fa93654","Type":"ContainerDied","Data":"0335c6314bf28f311ad7f9faf0ef791ab9a601cfedc88e8cfd7cdab6cd2c93b4"}
May 28 23:23:11.898973 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.898883    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs" event={"ID":"55638b90-c9f1-4397-986a-35ef1fa93654","Type":"ContainerDied","Data":"fbd0f42280c9b215805d651ae8738dd5dd37f776b126d2c897095471e934082c"}
May 28 23:23:11.898973 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.898898    2577 scope.go:117] "RemoveContainer" containerID="0335c6314bf28f311ad7f9faf0ef791ab9a601cfedc88e8cfd7cdab6cd2c93b4"
May 28 23:23:11.912593 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.912570    2577 scope.go:117] "RemoveContainer" containerID="0335c6314bf28f311ad7f9faf0ef791ab9a601cfedc88e8cfd7cdab6cd2c93b4"
May 28 23:23:11.913159 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:23:11.913129    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0335c6314bf28f311ad7f9faf0ef791ab9a601cfedc88e8cfd7cdab6cd2c93b4\": container with ID starting with 0335c6314bf28f311ad7f9faf0ef791ab9a601cfedc88e8cfd7cdab6cd2c93b4 not found: ID does not exist" containerID="0335c6314bf28f311ad7f9faf0ef791ab9a601cfedc88e8cfd7cdab6cd2c93b4"
May 28 23:23:11.913282 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.913166    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0335c6314bf28f311ad7f9faf0ef791ab9a601cfedc88e8cfd7cdab6cd2c93b4"} err="failed to get container status \"0335c6314bf28f311ad7f9faf0ef791ab9a601cfedc88e8cfd7cdab6cd2c93b4\": rpc error: code = NotFound desc = could not find container \"0335c6314bf28f311ad7f9faf0ef791ab9a601cfedc88e8cfd7cdab6cd2c93b4\": container with ID starting with 0335c6314bf28f311ad7f9faf0ef791ab9a601cfedc88e8cfd7cdab6cd2c93b4 not found: ID does not exist"
May 28 23:23:11.934198 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.934167    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs"]
May 28 23:23:11.946713 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.946683    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6865775464-kvvvs"]
May 28 23:23:11.977553 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:11.977520    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfhn2\" (UniqueName: \"kubernetes.io/projected/55638b90-c9f1-4397-986a-35ef1fa93654-kube-api-access-qfhn2\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:23:12.905400 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:12.905301    2577 generic.go:358] "Generic (PLEG): container finished" podID="89091c32-5e43-4c1c-8042-d5750a52b714" containerID="6fa069b1b9ba73b0d653ca5ff032097c81e77dfc4bcb0ad0e40932f888ccdd74" exitCode=0
May 28 23:23:12.905400 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:12.905374    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" event={"ID":"89091c32-5e43-4c1c-8042-d5750a52b714","Type":"ContainerDied","Data":"6fa069b1b9ba73b0d653ca5ff032097c81e77dfc4bcb0ad0e40932f888ccdd74"}
May 28 23:23:12.907724 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:12.907699    2577 generic.go:358] "Generic (PLEG): container finished" podID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerID="d4fcba655967655d1fbabe39d42cc75f071c7deddfbd864789033ea5896864e5" exitCode=0
May 28 23:23:12.907901 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:12.907779    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" event={"ID":"3c40243b-8f1a-496d-8082-7bd7be78391c","Type":"ContainerDied","Data":"d4fcba655967655d1fbabe39d42cc75f071c7deddfbd864789033ea5896864e5"}
May 28 23:23:13.492070 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:13.492031    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55638b90-c9f1-4397-986a-35ef1fa93654" path="/var/lib/kubelet/pods/55638b90-c9f1-4397-986a-35ef1fa93654/volumes"
May 28 23:23:14.073445 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.073410    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"]
May 28 23:23:14.073878 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.073683    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2" podUID="0be901ac-1cf0-45d2-98a8-fad87021c676" containerName="postgres" containerID="cri-o://00c4f897cf07bbd11bab966045847106920c888cb5c13a4d4e8e3432ce70933e" gracePeriod=30
May 28 23:23:14.078472 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.078442    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-kjdds"]
May 28 23:23:14.078698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.078673    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/redis-854544c579-kjdds" podUID="eb76630e-9170-4ccb-ac46-2c7adca56e4a" containerName="redis" containerID="cri-o://0fa73033209011385ac51945fc9687a1b1192b9ac7f4051563147f1a2afab1d1" gracePeriod=30
May 28 23:23:14.391920 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.391217    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-kjdds"
May 28 23:23:14.408889 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.397021    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"
May 28 23:23:14.501025 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.500993    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xz7m\" (UniqueName: \"kubernetes.io/projected/eb76630e-9170-4ccb-ac46-2c7adca56e4a-kube-api-access-2xz7m\") pod \"eb76630e-9170-4ccb-ac46-2c7adca56e4a\" (UID: \"eb76630e-9170-4ccb-ac46-2c7adca56e4a\") "
May 28 23:23:14.501166 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.501076    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/0be901ac-1cf0-45d2-98a8-fad87021c676-postgresdata\") pod \"0be901ac-1cf0-45d2-98a8-fad87021c676\" (UID: \"0be901ac-1cf0-45d2-98a8-fad87021c676\") "
May 28 23:23:14.501283 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.501254    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4trwq\" (UniqueName: \"kubernetes.io/projected/0be901ac-1cf0-45d2-98a8-fad87021c676-kube-api-access-4trwq\") pod \"0be901ac-1cf0-45d2-98a8-fad87021c676\" (UID: \"0be901ac-1cf0-45d2-98a8-fad87021c676\") "
May 28 23:23:14.503404 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.503374    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb76630e-9170-4ccb-ac46-2c7adca56e4a-kube-api-access-2xz7m" (OuterVolumeSpecName: "kube-api-access-2xz7m") pod "eb76630e-9170-4ccb-ac46-2c7adca56e4a" (UID: "eb76630e-9170-4ccb-ac46-2c7adca56e4a"). InnerVolumeSpecName "kube-api-access-2xz7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:23:14.503536 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.503395    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be901ac-1cf0-45d2-98a8-fad87021c676-kube-api-access-4trwq" (OuterVolumeSpecName: "kube-api-access-4trwq") pod "0be901ac-1cf0-45d2-98a8-fad87021c676" (UID: "0be901ac-1cf0-45d2-98a8-fad87021c676"). InnerVolumeSpecName "kube-api-access-4trwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:23:14.541770 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.541714    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be901ac-1cf0-45d2-98a8-fad87021c676-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "0be901ac-1cf0-45d2-98a8-fad87021c676" (UID: "0be901ac-1cf0-45d2-98a8-fad87021c676"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:23:14.602851 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.602813    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xz7m\" (UniqueName: \"kubernetes.io/projected/eb76630e-9170-4ccb-ac46-2c7adca56e4a-kube-api-access-2xz7m\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:23:14.602851 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.602848    2577 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/0be901ac-1cf0-45d2-98a8-fad87021c676-postgresdata\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:23:14.602851 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.602859    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4trwq\" (UniqueName: \"kubernetes.io/projected/0be901ac-1cf0-45d2-98a8-fad87021c676-kube-api-access-4trwq\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:23:14.917088 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.917041    2577 generic.go:358] "Generic (PLEG): container finished" podID="eb76630e-9170-4ccb-ac46-2c7adca56e4a" containerID="0fa73033209011385ac51945fc9687a1b1192b9ac7f4051563147f1a2afab1d1" exitCode=0
May 28 23:23:14.917273 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.917154    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-kjdds"
May 28 23:23:14.917273 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.917181    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-kjdds" event={"ID":"eb76630e-9170-4ccb-ac46-2c7adca56e4a","Type":"ContainerDied","Data":"0fa73033209011385ac51945fc9687a1b1192b9ac7f4051563147f1a2afab1d1"}
May 28 23:23:14.917273 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.917226    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-kjdds" event={"ID":"eb76630e-9170-4ccb-ac46-2c7adca56e4a","Type":"ContainerDied","Data":"7eccc48edffdc31f42fa373847edd2356e9aa9461b211b00c17bfd910c14ca14"}
May 28 23:23:14.917273 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.917246    2577 scope.go:117] "RemoveContainer" containerID="0fa73033209011385ac51945fc9687a1b1192b9ac7f4051563147f1a2afab1d1"
May 28 23:23:14.918658 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.918632    2577 generic.go:358] "Generic (PLEG): container finished" podID="0be901ac-1cf0-45d2-98a8-fad87021c676" containerID="00c4f897cf07bbd11bab966045847106920c888cb5c13a4d4e8e3432ce70933e" exitCode=0
May 28 23:23:14.918759 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.918680    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2" event={"ID":"0be901ac-1cf0-45d2-98a8-fad87021c676","Type":"ContainerDied","Data":"00c4f897cf07bbd11bab966045847106920c888cb5c13a4d4e8e3432ce70933e"}
May 28 23:23:14.918759 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.918697    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"
May 28 23:23:14.918759 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.918706    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-lrhg2" event={"ID":"0be901ac-1cf0-45d2-98a8-fad87021c676","Type":"ContainerDied","Data":"e652cf43caebd1aed59a7c31153fadb9b3263e474b4b07fb11a2fbdfe628362b"}
May 28 23:23:14.927617 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.927600    2577 scope.go:117] "RemoveContainer" containerID="0fa73033209011385ac51945fc9687a1b1192b9ac7f4051563147f1a2afab1d1"
May 28 23:23:14.927979 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:23:14.927945    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa73033209011385ac51945fc9687a1b1192b9ac7f4051563147f1a2afab1d1\": container with ID starting with 0fa73033209011385ac51945fc9687a1b1192b9ac7f4051563147f1a2afab1d1 not found: ID does not exist" containerID="0fa73033209011385ac51945fc9687a1b1192b9ac7f4051563147f1a2afab1d1"
May 28 23:23:14.928073 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.927985    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa73033209011385ac51945fc9687a1b1192b9ac7f4051563147f1a2afab1d1"} err="failed to get container status \"0fa73033209011385ac51945fc9687a1b1192b9ac7f4051563147f1a2afab1d1\": rpc error: code = NotFound desc = could not find container \"0fa73033209011385ac51945fc9687a1b1192b9ac7f4051563147f1a2afab1d1\": container with ID starting with 0fa73033209011385ac51945fc9687a1b1192b9ac7f4051563147f1a2afab1d1 not found: ID does not exist"
May 28 23:23:14.928073 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.928011    2577 scope.go:117] "RemoveContainer" containerID="00c4f897cf07bbd11bab966045847106920c888cb5c13a4d4e8e3432ce70933e"
May 28 23:23:14.936666 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.936640    2577 scope.go:117] "RemoveContainer" containerID="00c4f897cf07bbd11bab966045847106920c888cb5c13a4d4e8e3432ce70933e"
May 28 23:23:14.936974 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:23:14.936954    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c4f897cf07bbd11bab966045847106920c888cb5c13a4d4e8e3432ce70933e\": container with ID starting with 00c4f897cf07bbd11bab966045847106920c888cb5c13a4d4e8e3432ce70933e not found: ID does not exist" containerID="00c4f897cf07bbd11bab966045847106920c888cb5c13a4d4e8e3432ce70933e"
May 28 23:23:14.937039 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.936982    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c4f897cf07bbd11bab966045847106920c888cb5c13a4d4e8e3432ce70933e"} err="failed to get container status \"00c4f897cf07bbd11bab966045847106920c888cb5c13a4d4e8e3432ce70933e\": rpc error: code = NotFound desc = could not find container \"00c4f897cf07bbd11bab966045847106920c888cb5c13a4d4e8e3432ce70933e\": container with ID starting with 00c4f897cf07bbd11bab966045847106920c888cb5c13a4d4e8e3432ce70933e not found: ID does not exist"
May 28 23:23:14.940181 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.940154    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-kjdds"]
May 28 23:23:14.943656 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.943635    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-854544c579-kjdds"]
May 28 23:23:14.956298 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.956251    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"]
May 28 23:23:14.959022 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:14.958983    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-lrhg2"]
May 28 23:23:15.270798 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:15.270759    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused"
May 28 23:23:15.495585 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:15.495552    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0be901ac-1cf0-45d2-98a8-fad87021c676" path="/var/lib/kubelet/pods/0be901ac-1cf0-45d2-98a8-fad87021c676/volumes"
May 28 23:23:15.496089 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:15.496066    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb76630e-9170-4ccb-ac46-2c7adca56e4a" path="/var/lib/kubelet/pods/eb76630e-9170-4ccb-ac46-2c7adca56e4a/volumes"
May 28 23:23:21.744405 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:21.744359    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused"
May 28 23:23:25.270944 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:25.270856    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused"
May 28 23:23:31.744134 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:31.744088    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused"
May 28 23:23:35.271113 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:35.271066    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused"
May 28 23:23:35.271515 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:35.271192    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:23:41.745191 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:41.745150    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused"
May 28 23:23:41.745589 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:41.745297    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:23:42.013233 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.013199    2577 generic.go:358] "Generic (PLEG): container finished" podID="89091c32-5e43-4c1c-8042-d5750a52b714" containerID="89dd3741e8781c359345d82a6d876466fb4df1800d642677c138d8bda6d37299" exitCode=137
May 28 23:23:42.013407 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.013278    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" event={"ID":"89091c32-5e43-4c1c-8042-d5750a52b714","Type":"ContainerDied","Data":"89dd3741e8781c359345d82a6d876466fb4df1800d642677c138d8bda6d37299"}
May 28 23:23:42.015299 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.015275    2577 generic.go:358] "Generic (PLEG): container finished" podID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerID="264b7e76f1f092b16ca112ac248e36a5ca31903b3eff13ac9e85392f940608f9" exitCode=137
May 28 23:23:42.015405 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.015345    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" event={"ID":"3c40243b-8f1a-496d-8082-7bd7be78391c","Type":"ContainerDied","Data":"264b7e76f1f092b16ca112ac248e36a5ca31903b3eff13ac9e85392f940608f9"}
May 28 23:23:42.350154 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.350130    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:23:42.383187 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.383164    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:23:42.452372 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.452340    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3c40243b-8f1a-496d-8082-7bd7be78391c-registry-tls\") pod \"3c40243b-8f1a-496d-8082-7bd7be78391c\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") "
May 28 23:23:42.452537 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.452380    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3c40243b-8f1a-496d-8082-7bd7be78391c-online-tls\") pod \"3c40243b-8f1a-496d-8082-7bd7be78391c\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") "
May 28 23:23:42.452537 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.452404    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/89091c32-5e43-4c1c-8042-d5750a52b714-feast-data\") pod \"89091c32-5e43-4c1c-8042-d5750a52b714\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") "
May 28 23:23:42.452646 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.452529    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3c40243b-8f1a-496d-8082-7bd7be78391c-feast-data\") pod \"3c40243b-8f1a-496d-8082-7bd7be78391c\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") "
May 28 23:23:42.452646 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.452589    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2wck\" (UniqueName: \"kubernetes.io/projected/89091c32-5e43-4c1c-8042-d5750a52b714-kube-api-access-s2wck\") pod \"89091c32-5e43-4c1c-8042-d5750a52b714\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") "
May 28 23:23:42.452646 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.452622    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89091c32-5e43-4c1c-8042-d5750a52b714-online-tls\") pod \"89091c32-5e43-4c1c-8042-d5750a52b714\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") "
May 28 23:23:42.452803 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.452666    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/89091c32-5e43-4c1c-8042-d5750a52b714-registry-tls\") pod \"89091c32-5e43-4c1c-8042-d5750a52b714\" (UID: \"89091c32-5e43-4c1c-8042-d5750a52b714\") "
May 28 23:23:42.452803 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.452697    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xg9p\" (UniqueName: \"kubernetes.io/projected/3c40243b-8f1a-496d-8082-7bd7be78391c-kube-api-access-9xg9p\") pod \"3c40243b-8f1a-496d-8082-7bd7be78391c\" (UID: \"3c40243b-8f1a-496d-8082-7bd7be78391c\") "
May 28 23:23:42.452937 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.452902    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89091c32-5e43-4c1c-8042-d5750a52b714-feast-data" (OuterVolumeSpecName: "feast-data") pod "89091c32-5e43-4c1c-8042-d5750a52b714" (UID: "89091c32-5e43-4c1c-8042-d5750a52b714"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:23:42.453078 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.453046    2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/89091c32-5e43-4c1c-8042-d5750a52b714-feast-data\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:23:42.454856 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.454827    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c40243b-8f1a-496d-8082-7bd7be78391c-online-tls" (OuterVolumeSpecName: "online-tls") pod "3c40243b-8f1a-496d-8082-7bd7be78391c" (UID: "3c40243b-8f1a-496d-8082-7bd7be78391c"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:23:42.455346 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.455318    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c40243b-8f1a-496d-8082-7bd7be78391c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3c40243b-8f1a-496d-8082-7bd7be78391c" (UID: "3c40243b-8f1a-496d-8082-7bd7be78391c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:23:42.455458 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.455425    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c40243b-8f1a-496d-8082-7bd7be78391c-kube-api-access-9xg9p" (OuterVolumeSpecName: "kube-api-access-9xg9p") pod "3c40243b-8f1a-496d-8082-7bd7be78391c" (UID: "3c40243b-8f1a-496d-8082-7bd7be78391c"). InnerVolumeSpecName "kube-api-access-9xg9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:23:42.455522 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.455504    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89091c32-5e43-4c1c-8042-d5750a52b714-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "89091c32-5e43-4c1c-8042-d5750a52b714" (UID: "89091c32-5e43-4c1c-8042-d5750a52b714"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:23:42.455576 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.455528    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89091c32-5e43-4c1c-8042-d5750a52b714-online-tls" (OuterVolumeSpecName: "online-tls") pod "89091c32-5e43-4c1c-8042-d5750a52b714" (UID: "89091c32-5e43-4c1c-8042-d5750a52b714"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:23:42.455629 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.455600    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89091c32-5e43-4c1c-8042-d5750a52b714-kube-api-access-s2wck" (OuterVolumeSpecName: "kube-api-access-s2wck") pod "89091c32-5e43-4c1c-8042-d5750a52b714" (UID: "89091c32-5e43-4c1c-8042-d5750a52b714"). InnerVolumeSpecName "kube-api-access-s2wck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:23:42.462406 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.462379    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c40243b-8f1a-496d-8082-7bd7be78391c-feast-data" (OuterVolumeSpecName: "feast-data") pod "3c40243b-8f1a-496d-8082-7bd7be78391c" (UID: "3c40243b-8f1a-496d-8082-7bd7be78391c"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:23:42.553750 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.553674    2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3c40243b-8f1a-496d-8082-7bd7be78391c-feast-data\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:23:42.553750 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.553704    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2wck\" (UniqueName: \"kubernetes.io/projected/89091c32-5e43-4c1c-8042-d5750a52b714-kube-api-access-s2wck\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:23:42.553750 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.553719    2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/89091c32-5e43-4c1c-8042-d5750a52b714-online-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:23:42.553750 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.553732    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/89091c32-5e43-4c1c-8042-d5750a52b714-registry-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:23:42.553750 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.553744    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xg9p\" (UniqueName: \"kubernetes.io/projected/3c40243b-8f1a-496d-8082-7bd7be78391c-kube-api-access-9xg9p\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:23:42.554090 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.553758    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3c40243b-8f1a-496d-8082-7bd7be78391c-registry-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:23:42.554090 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:42.553771    2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3c40243b-8f1a-496d-8082-7bd7be78391c-online-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:23:43.023341 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.023303    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l" event={"ID":"89091c32-5e43-4c1c-8042-d5750a52b714","Type":"ContainerDied","Data":"21e99333fff7de879a61ff02a2e4fb5fb018ff0630e65844b26a56de5fb5ef15"}
May 28 23:23:43.023829 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.023346    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"
May 28 23:23:43.023829 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.023360    2577 scope.go:117] "RemoveContainer" containerID="6fa069b1b9ba73b0d653ca5ff032097c81e77dfc4bcb0ad0e40932f888ccdd74"
May 28 23:23:43.025581 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.025549    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2" event={"ID":"3c40243b-8f1a-496d-8082-7bd7be78391c","Type":"ContainerDied","Data":"1540392982f1d31fef6de0804e2ce995338ec21415ede29ef53b36bd56e18d93"}
May 28 23:23:43.025581 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.025574    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"
May 28 23:23:43.033279 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.033258    2577 scope.go:117] "RemoveContainer" containerID="89dd3741e8781c359345d82a6d876466fb4df1800d642677c138d8bda6d37299"
May 28 23:23:43.041547 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.041528    2577 scope.go:117] "RemoveContainer" containerID="e0fd4e98d1c00a50e89453f744ba52b71efc3a81d66073cbc58da7f35ed120a7"
May 28 23:23:43.049339 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.049315    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"]
May 28 23:23:43.050250 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.050216    2577 scope.go:117] "RemoveContainer" containerID="b8b3ea8a53307db50e01fb1edbe98adff715db26f03e6437f51d48e4a75470a9"
May 28 23:23:43.052200 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.052178    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-5d85d8b884-xfn6l"]
May 28 23:23:43.057845 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.057826    2577 scope.go:117] "RemoveContainer" containerID="d4fcba655967655d1fbabe39d42cc75f071c7deddfbd864789033ea5896864e5"
May 28 23:23:43.063491 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.063469    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"]
May 28 23:23:43.065660 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.065644    2577 scope.go:117] "RemoveContainer" containerID="264b7e76f1f092b16ca112ac248e36a5ca31903b3eff13ac9e85392f940608f9"
May 28 23:23:43.068843 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.068820    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-65b7c7ff87-c5bc2"]
May 28 23:23:43.073650 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.073633    2577 scope.go:117] "RemoveContainer" containerID="ec1e4336477ecd7973fa2147e818655ac97ba6a69a3bbc2e1c9c57f75cf698ef"
May 28 23:23:43.080635 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.080618    2577 scope.go:117] "RemoveContainer" containerID="7cc7741ad7fcd7c2c607423c13198257bcb916ad282ec9c10b111da0b44f95b6"
May 28 23:23:43.494546 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.494513    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" path="/var/lib/kubelet/pods/3c40243b-8f1a-496d-8082-7bd7be78391c/volumes"
May 28 23:23:43.495051 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:43.495037    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" path="/var/lib/kubelet/pods/89091c32-5e43-4c1c-8042-d5750a52b714/volumes"
May 28 23:23:58.254324 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254287    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"]
May 28 23:23:58.254837 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254820    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="registry"
May 28 23:23:58.254935 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254841    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="registry"
May 28 23:23:58.254935 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254879    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="registry"
May 28 23:23:58.254935 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254889    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="registry"
May 28 23:23:58.254935 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254899    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="feast-init"
May 28 23:23:58.254935 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254906    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="feast-init"
May 28 23:23:58.254935 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254923    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb76630e-9170-4ccb-ac46-2c7adca56e4a" containerName="redis"
May 28 23:23:58.254935 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254931    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb76630e-9170-4ccb-ac46-2c7adca56e4a" containerName="redis"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254943    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="feast-init"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254952    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="feast-init"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254962    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="feast-apply"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254970    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="feast-apply"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254986    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="feast-apply"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254991    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="feast-apply"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.254998    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="online"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.255003    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="online"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.255017    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55638b90-c9f1-4397-986a-35ef1fa93654" containerName="manager"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.255025    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="55638b90-c9f1-4397-986a-35ef1fa93654" containerName="manager"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.255035    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be901ac-1cf0-45d2-98a8-fad87021c676" containerName="postgres"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.255039    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be901ac-1cf0-45d2-98a8-fad87021c676" containerName="postgres"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.255044    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="online"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.255049    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="online"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.255124    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="eb76630e-9170-4ccb-ac46-2c7adca56e4a" containerName="redis"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.255133    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="0be901ac-1cf0-45d2-98a8-fad87021c676" containerName="postgres"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.255140    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="online"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.255147    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="3c40243b-8f1a-496d-8082-7bd7be78391c" containerName="registry"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.255154    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="online"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.255162    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="55638b90-c9f1-4397-986a-35ef1fa93654" containerName="manager"
May 28 23:23:58.255293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.255167    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="89091c32-5e43-4c1c-8042-d5750a52b714" containerName="registry"
May 28 23:23:58.260907 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.260889    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.263654 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.263629    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-pgmsf\""
May 28 23:23:58.263791 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.263672    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 28 23:23:58.263791 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.263629    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 23:23:58.263791 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.263723    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 28 23:23:58.264492 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.264477    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 28 23:23:58.264571 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.264480    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 28 23:23:58.264624 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.264482    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 23:23:58.269281 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.269257    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"]
May 28 23:23:58.391552 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.391513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.391747 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.391565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.391747 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.391666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.391747 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.391702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e2715e5d-f90a-46be-883b-05d36da15d82-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.391747 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.391735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.391986 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.391757    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpdgj\" (UniqueName: \"kubernetes.io/projected/e2715e5d-f90a-46be-883b-05d36da15d82-kube-api-access-fpdgj\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.492343 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.492309    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.492343 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.492347    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e2715e5d-f90a-46be-883b-05d36da15d82-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.492608 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.492368    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.492608 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.492388    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fpdgj\" (UniqueName: \"kubernetes.io/projected/e2715e5d-f90a-46be-883b-05d36da15d82-kube-api-access-fpdgj\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.492608 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.492417    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.492608 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.492474    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.492608 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:23:58.492577    2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 28 23:23:58.492905 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:23:58.492652    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-registry-tls podName:e2715e5d-f90a-46be-883b-05d36da15d82 nodeName:}" failed. No retries permitted until 2026-05-28 23:23:58.992630326 +0000 UTC m=+1312.186061561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-registry-tls") pod "feast-simple-feast-setup-664fcfcc58-g8dc9" (UID: "e2715e5d-f90a-46be-883b-05d36da15d82") : secret "feast-simple-feast-setup-registry-tls" not found
May 28 23:23:58.492905 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.492760    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e2715e5d-f90a-46be-883b-05d36da15d82-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.495170 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.495146    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.495170 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.495164    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.495297 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.495155    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.504492 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.504429    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpdgj\" (UniqueName: \"kubernetes.io/projected/e2715e5d-f90a-46be-883b-05d36da15d82-kube-api-access-fpdgj\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:58.997507 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.997478    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:59.000031 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:58.999997    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-g8dc9\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:59.185343 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:59.185295    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:23:59.321560 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:59.321537    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"]
May 28 23:23:59.323211 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:23:59.323180    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2715e5d_f90a_46be_883b_05d36da15d82.slice/crio-42b5bce58cfa9d5049e0e6e55bc80be2baf97dd736519e45fd7e4e41e0c3f203 WatchSource:0}: Error finding container 42b5bce58cfa9d5049e0e6e55bc80be2baf97dd736519e45fd7e4e41e0c3f203: Status 404 returned error can't find the container with id 42b5bce58cfa9d5049e0e6e55bc80be2baf97dd736519e45fd7e4e41e0c3f203
May 28 23:23:59.324838 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:23:59.324823    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 23:24:00.090668 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:00.090625    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" event={"ID":"e2715e5d-f90a-46be-883b-05d36da15d82","Type":"ContainerStarted","Data":"42b5bce58cfa9d5049e0e6e55bc80be2baf97dd736519e45fd7e4e41e0c3f203"}
May 28 23:24:15.150370 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:15.150333    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" event={"ID":"e2715e5d-f90a-46be-883b-05d36da15d82","Type":"ContainerStarted","Data":"5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06"}
May 28 23:24:19.165757 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:19.165673    2577 generic.go:358] "Generic (PLEG): container finished" podID="e2715e5d-f90a-46be-883b-05d36da15d82" containerID="5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06" exitCode=0
May 28 23:24:19.166129 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:19.165748    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" event={"ID":"e2715e5d-f90a-46be-883b-05d36da15d82","Type":"ContainerDied","Data":"5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06"}
May 28 23:24:20.172068 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:20.172029    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" event={"ID":"e2715e5d-f90a-46be-883b-05d36da15d82","Type":"ContainerStarted","Data":"99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71"}
May 28 23:24:25.197738 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:25.197701    2577 generic.go:358] "Generic (PLEG): container finished" podID="e2715e5d-f90a-46be-883b-05d36da15d82" containerID="99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71" exitCode=0
May 28 23:24:25.198138 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:25.197774    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" event={"ID":"e2715e5d-f90a-46be-883b-05d36da15d82","Type":"ContainerDied","Data":"99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71"}
May 28 23:24:26.205645 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:26.205601    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" event={"ID":"e2715e5d-f90a-46be-883b-05d36da15d82","Type":"ContainerStarted","Data":"175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210"}
May 28 23:24:26.206160 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:26.205660    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" event={"ID":"e2715e5d-f90a-46be-883b-05d36da15d82","Type":"ContainerStarted","Data":"cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095"}
May 28 23:24:26.206160 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:26.205677    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" event={"ID":"e2715e5d-f90a-46be-883b-05d36da15d82","Type":"ContainerStarted","Data":"b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312"}
May 28 23:24:26.206160 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:26.205690    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" event={"ID":"e2715e5d-f90a-46be-883b-05d36da15d82","Type":"ContainerStarted","Data":"b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c"}
May 28 23:24:26.233750 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:26.233682    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" podStartSLOduration=13.37163488 podStartE2EDuration="28.23366054s" podCreationTimestamp="2026-05-28 23:23:58 +0000 UTC" firstStartedPulling="2026-05-28 23:23:59.324973788 +0000 UTC m=+1312.518405018" lastFinishedPulling="2026-05-28 23:24:14.186999436 +0000 UTC m=+1327.380430678" observedRunningTime="2026-05-28 23:24:26.230173312 +0000 UTC m=+1339.423604564" watchObservedRunningTime="2026-05-28 23:24:26.23366054 +0000 UTC m=+1339.427091793"
May 28 23:24:29.186609 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:29.186559    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:24:29.187246 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:29.187224    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:24:29.187385 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:29.187371    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:24:29.188011 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:29.187987    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:24:29.188143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:29.188129    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:24:29.188253 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:29.188230    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:24:29.188253 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:29.188255    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:24:29.188438 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:29.188267    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:24:29.188855 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:29.188822    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused"
May 28 23:24:29.189039 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:29.188849    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.40:8816: connect: connection refused"
May 28 23:24:29.189118 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:29.188918    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.40:6571: connect: connection refused"
May 28 23:24:29.189118 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:29.188905    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused"
May 28 23:24:32.186999 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:32.186969    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:24:32.187370 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:32.187133    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:24:32.187370 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:32.187312    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:24:32.187888 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:32.187838    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:24:32.188015 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:32.187923    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:24:32.188151 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:32.188132    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:24:32.191499 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:32.191481    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:24:32.238542 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:24:32.238508    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:25:34.625367 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:34.625323    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"]
May 28 23:25:34.625877 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:34.625810    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="registry" containerID="cri-o://b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c" gracePeriod=30
May 28 23:25:34.625978 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:34.625854    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="ui" containerID="cri-o://175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210" gracePeriod=30
May 28 23:25:34.625978 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:34.625903    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="online" containerID="cri-o://b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312" gracePeriod=30
May 28 23:25:34.625978 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:34.625855    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="offline" containerID="cri-o://cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095" gracePeriod=30
May 28 23:25:34.919241 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:34.919136    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"]
May 28 23:25:34.929121 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:34.929094    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:34.934490 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:34.934464    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-56666\""
May 28 23:25:34.937079 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:34.937049    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"]
May 28 23:25:35.049781 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.049744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.049999 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.049815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bsrw\" (UniqueName: \"kubernetes.io/projected/027df7a5-98d8-47a2-8959-80ac6dbba4ec-kube-api-access-5bsrw\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.049999 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.049850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.049999 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.049949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.050171 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.050021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.050171 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.050043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/027df7a5-98d8-47a2-8959-80ac6dbba4ec-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.151262 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.151227    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5bsrw\" (UniqueName: \"kubernetes.io/projected/027df7a5-98d8-47a2-8959-80ac6dbba4ec-kube-api-access-5bsrw\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.151434 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.151281    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.151434 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.151321    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.151434 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.151390    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.151434 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.151411    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/027df7a5-98d8-47a2-8959-80ac6dbba4ec-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.151644 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.151464    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.151644 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:25:35.151597    2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 28 23:25:35.151741 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:25:35.151669    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-online-tls podName:027df7a5-98d8-47a2-8959-80ac6dbba4ec nodeName:}" failed. No retries permitted until 2026-05-28 23:25:35.651646019 +0000 UTC m=+1408.845077264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-online-tls") pod "feast-simple-feast-setup-664fcfcc58-fxgkt" (UID: "027df7a5-98d8-47a2-8959-80ac6dbba4ec") : secret "feast-simple-feast-setup-online-tls" not found
May 28 23:25:35.152020 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:25:35.151997    2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 28 23:25:35.152144 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:25:35.152056    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-ui-tls podName:027df7a5-98d8-47a2-8959-80ac6dbba4ec nodeName:}" failed. No retries permitted until 2026-05-28 23:25:35.652038324 +0000 UTC m=+1408.845469554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-ui-tls") pod "feast-simple-feast-setup-664fcfcc58-fxgkt" (UID: "027df7a5-98d8-47a2-8959-80ac6dbba4ec") : secret "feast-simple-feast-setup-ui-tls" not found
May 28 23:25:35.152144 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:25:35.151997    2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 28 23:25:35.152144 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:25:35.152097    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-registry-tls podName:027df7a5-98d8-47a2-8959-80ac6dbba4ec nodeName:}" failed. No retries permitted until 2026-05-28 23:25:35.652087597 +0000 UTC m=+1408.845518828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-registry-tls") pod "feast-simple-feast-setup-664fcfcc58-fxgkt" (UID: "027df7a5-98d8-47a2-8959-80ac6dbba4ec") : secret "feast-simple-feast-setup-registry-tls" not found
May 28 23:25:35.152313 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.152292    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/027df7a5-98d8-47a2-8959-80ac6dbba4ec-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.154479 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.154450    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.160821 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.160792    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bsrw\" (UniqueName: \"kubernetes.io/projected/027df7a5-98d8-47a2-8959-80ac6dbba4ec-kube-api-access-5bsrw\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.442658 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.442626    2577 generic.go:358] "Generic (PLEG): container finished" podID="e2715e5d-f90a-46be-883b-05d36da15d82" containerID="175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210" exitCode=0
May 28 23:25:35.442658 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.442653    2577 generic.go:358] "Generic (PLEG): container finished" podID="e2715e5d-f90a-46be-883b-05d36da15d82" containerID="b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312" exitCode=0
May 28 23:25:35.442852 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.442702    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" event={"ID":"e2715e5d-f90a-46be-883b-05d36da15d82","Type":"ContainerDied","Data":"175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210"}
May 28 23:25:35.442852 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.442737    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" event={"ID":"e2715e5d-f90a-46be-883b-05d36da15d82","Type":"ContainerDied","Data":"b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312"}
May 28 23:25:35.656297 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.656208    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.656747 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.656304    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.656747 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.656358    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.658716 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.658694    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.658816 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.658797    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.658966 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.658946    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-fxgkt\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.849541 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.849499    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:35.985119 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:35.985093    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"]
May 28 23:25:35.987259 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:25:35.987231    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod027df7a5_98d8_47a2_8959_80ac6dbba4ec.slice/crio-2857e4d8fc9c59a7f4d2eb10a8b05e716d6d371120b47f6241f590312fff8768 WatchSource:0}: Error finding container 2857e4d8fc9c59a7f4d2eb10a8b05e716d6d371120b47f6241f590312fff8768: Status 404 returned error can't find the container with id 2857e4d8fc9c59a7f4d2eb10a8b05e716d6d371120b47f6241f590312fff8768
May 28 23:25:36.447975 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:36.447928    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" event={"ID":"027df7a5-98d8-47a2-8959-80ac6dbba4ec","Type":"ContainerStarted","Data":"4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74"}
May 28 23:25:36.447975 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:36.447975    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" event={"ID":"027df7a5-98d8-47a2-8959-80ac6dbba4ec","Type":"ContainerStarted","Data":"2857e4d8fc9c59a7f4d2eb10a8b05e716d6d371120b47f6241f590312fff8768"}
May 28 23:25:41.467510 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:41.467472    2577 generic.go:358] "Generic (PLEG): container finished" podID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerID="4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74" exitCode=0
May 28 23:25:41.467892 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:41.467542    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" event={"ID":"027df7a5-98d8-47a2-8959-80ac6dbba4ec","Type":"ContainerDied","Data":"4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74"}
May 28 23:25:42.188206 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:42.188156    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused"
May 28 23:25:42.235668 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:42.235626    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused"
May 28 23:25:42.473278 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:42.473190    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" event={"ID":"027df7a5-98d8-47a2-8959-80ac6dbba4ec","Type":"ContainerStarted","Data":"4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d"}
May 28 23:25:47.492693 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:47.492661    2577 generic.go:358] "Generic (PLEG): container finished" podID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerID="4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d" exitCode=0
May 28 23:25:47.493098 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:47.492735    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" event={"ID":"027df7a5-98d8-47a2-8959-80ac6dbba4ec","Type":"ContainerDied","Data":"4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d"}
May 28 23:25:48.500670 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:48.500619    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" event={"ID":"027df7a5-98d8-47a2-8959-80ac6dbba4ec","Type":"ContainerStarted","Data":"9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e"}
May 28 23:25:48.501278 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:48.501239    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" event={"ID":"027df7a5-98d8-47a2-8959-80ac6dbba4ec","Type":"ContainerStarted","Data":"639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc"}
May 28 23:25:48.501278 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:48.501291    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" event={"ID":"027df7a5-98d8-47a2-8959-80ac6dbba4ec","Type":"ContainerStarted","Data":"df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35"}
May 28 23:25:48.501478 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:48.501306    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" event={"ID":"027df7a5-98d8-47a2-8959-80ac6dbba4ec","Type":"ContainerStarted","Data":"c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478"}
May 28 23:25:48.533846 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:48.532552    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" podStartSLOduration=14.532533279 podStartE2EDuration="14.532533279s" podCreationTimestamp="2026-05-28 23:25:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:25:48.529226525 +0000 UTC m=+1421.722657780" watchObservedRunningTime="2026-05-28 23:25:48.532533279 +0000 UTC m=+1421.725964532"
May 28 23:25:50.850345 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:50.850254    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:50.850345 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:50.850303    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:50.850345 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:50.850317    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:50.850345 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:50.850331    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:50.852351 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:50.852316    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused"
May 28 23:25:50.852607 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:50.852319    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused"
May 28 23:25:50.852718 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:50.852336    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.41:8816: connect: connection refused"
May 28 23:25:50.852718 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:50.852387    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.41:6571: connect: connection refused"
May 28 23:25:52.188274 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:52.188222    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused"
May 28 23:25:52.235756 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:52.235708    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused"
May 28 23:25:53.851174 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:53.851140    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:53.851599 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:53.851296    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:53.851599 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:53.851504    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:53.851839 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:53.851807    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:53.852090 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:53.851848    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:53.852090 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:53.851880    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:53.852090 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:53.851996    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:53.852230 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:53.852187    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:53.852308 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:53.852291    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:53.855844 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:53.855827    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:54.523801 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:54.523768    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:25:54.526733 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:25:54.526709    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:26:02.188491 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:02.188445    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused"
May 28 23:26:02.188951 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:02.188579    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:26:02.235222 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:02.235184    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused"
May 28 23:26:02.235370 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:02.235285    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:26:05.271545 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.271522    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:26:05.309066 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.309032    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-online-tls\") pod \"e2715e5d-f90a-46be-883b-05d36da15d82\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") "
May 28 23:26:05.309212 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.309098    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-ui-tls\") pod \"e2715e5d-f90a-46be-883b-05d36da15d82\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") "
May 28 23:26:05.309212 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.309130    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e2715e5d-f90a-46be-883b-05d36da15d82-feast-data\") pod \"e2715e5d-f90a-46be-883b-05d36da15d82\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") "
May 28 23:26:05.309212 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.309182    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-registry-tls\") pod \"e2715e5d-f90a-46be-883b-05d36da15d82\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") "
May 28 23:26:05.309371 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.309214    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-offline-tls\") pod \"e2715e5d-f90a-46be-883b-05d36da15d82\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") "
May 28 23:26:05.309371 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.309269    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpdgj\" (UniqueName: \"kubernetes.io/projected/e2715e5d-f90a-46be-883b-05d36da15d82-kube-api-access-fpdgj\") pod \"e2715e5d-f90a-46be-883b-05d36da15d82\" (UID: \"e2715e5d-f90a-46be-883b-05d36da15d82\") "
May 28 23:26:05.309848 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.309720    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2715e5d-f90a-46be-883b-05d36da15d82-feast-data" (OuterVolumeSpecName: "feast-data") pod "e2715e5d-f90a-46be-883b-05d36da15d82" (UID: "e2715e5d-f90a-46be-883b-05d36da15d82"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:26:05.311612 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.311580    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-online-tls" (OuterVolumeSpecName: "online-tls") pod "e2715e5d-f90a-46be-883b-05d36da15d82" (UID: "e2715e5d-f90a-46be-883b-05d36da15d82"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:26:05.311612 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.311591    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e2715e5d-f90a-46be-883b-05d36da15d82" (UID: "e2715e5d-f90a-46be-883b-05d36da15d82"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:26:05.311799 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.311741    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e2715e5d-f90a-46be-883b-05d36da15d82" (UID: "e2715e5d-f90a-46be-883b-05d36da15d82"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:26:05.312044 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.312024    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e2715e5d-f90a-46be-883b-05d36da15d82" (UID: "e2715e5d-f90a-46be-883b-05d36da15d82"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:26:05.312119 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.312092    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2715e5d-f90a-46be-883b-05d36da15d82-kube-api-access-fpdgj" (OuterVolumeSpecName: "kube-api-access-fpdgj") pod "e2715e5d-f90a-46be-883b-05d36da15d82" (UID: "e2715e5d-f90a-46be-883b-05d36da15d82"). InnerVolumeSpecName "kube-api-access-fpdgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:26:05.410792 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.410719    2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-online-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:26:05.410792 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.410745    2577 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-ui-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:26:05.410792 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.410754    2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e2715e5d-f90a-46be-883b-05d36da15d82-feast-data\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:26:05.410792 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.410762    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-registry-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:26:05.410792 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.410769    2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e2715e5d-f90a-46be-883b-05d36da15d82-offline-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:26:05.410792 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.410778    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpdgj\" (UniqueName: \"kubernetes.io/projected/e2715e5d-f90a-46be-883b-05d36da15d82-kube-api-access-fpdgj\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:26:05.562517 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.562487    2577 generic.go:358] "Generic (PLEG): container finished" podID="e2715e5d-f90a-46be-883b-05d36da15d82" containerID="cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095" exitCode=137
May 28 23:26:05.562517 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.562513    2577 generic.go:358] "Generic (PLEG): container finished" podID="e2715e5d-f90a-46be-883b-05d36da15d82" containerID="b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c" exitCode=137
May 28 23:26:05.562760 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.562556    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" event={"ID":"e2715e5d-f90a-46be-883b-05d36da15d82","Type":"ContainerDied","Data":"cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095"}
May 28 23:26:05.562760 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.562575    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"
May 28 23:26:05.562760 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.562609    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" event={"ID":"e2715e5d-f90a-46be-883b-05d36da15d82","Type":"ContainerDied","Data":"b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c"}
May 28 23:26:05.562760 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.562626    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9" event={"ID":"e2715e5d-f90a-46be-883b-05d36da15d82","Type":"ContainerDied","Data":"42b5bce58cfa9d5049e0e6e55bc80be2baf97dd736519e45fd7e4e41e0c3f203"}
May 28 23:26:05.562760 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.562645    2577 scope.go:117] "RemoveContainer" containerID="175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210"
May 28 23:26:05.571536 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.571521    2577 scope.go:117] "RemoveContainer" containerID="cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095"
May 28 23:26:05.579079 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.579053    2577 scope.go:117] "RemoveContainer" containerID="b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312"
May 28 23:26:05.584431 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.584408    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"]
May 28 23:26:05.587895 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.587853    2577 scope.go:117] "RemoveContainer" containerID="b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c"
May 28 23:26:05.588000 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.587980    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-g8dc9"]
May 28 23:26:05.595419 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.595401    2577 scope.go:117] "RemoveContainer" containerID="99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71"
May 28 23:26:05.602397 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.602375    2577 scope.go:117] "RemoveContainer" containerID="5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06"
May 28 23:26:05.609200 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.609181    2577 scope.go:117] "RemoveContainer" containerID="175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210"
May 28 23:26:05.609456 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:26:05.609438    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210\": container with ID starting with 175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210 not found: ID does not exist" containerID="175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210"
May 28 23:26:05.609504 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.609463    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210"} err="failed to get container status \"175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210\": rpc error: code = NotFound desc = could not find container \"175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210\": container with ID starting with 175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210 not found: ID does not exist"
May 28 23:26:05.609504 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.609482    2577 scope.go:117] "RemoveContainer" containerID="cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095"
May 28 23:26:05.609699 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:26:05.609680    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095\": container with ID starting with cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095 not found: ID does not exist" containerID="cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095"
May 28 23:26:05.609741 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.609705    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095"} err="failed to get container status \"cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095\": rpc error: code = NotFound desc = could not find container \"cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095\": container with ID starting with cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095 not found: ID does not exist"
May 28 23:26:05.609741 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.609717    2577 scope.go:117] "RemoveContainer" containerID="b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312"
May 28 23:26:05.609960 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:26:05.609945    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312\": container with ID starting with b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312 not found: ID does not exist" containerID="b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312"
May 28 23:26:05.610004 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.609963    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312"} err="failed to get container status \"b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312\": rpc error: code = NotFound desc = could not find container \"b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312\": container with ID starting with b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312 not found: ID does not exist"
May 28 23:26:05.610004 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.609976    2577 scope.go:117] "RemoveContainer" containerID="b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c"
May 28 23:26:05.610212 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:26:05.610195    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c\": container with ID starting with b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c not found: ID does not exist" containerID="b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c"
May 28 23:26:05.610255 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.610221    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c"} err="failed to get container status \"b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c\": rpc error: code = NotFound desc = could not find container \"b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c\": container with ID starting with b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c not found: ID does not exist"
May 28 23:26:05.610255 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.610240    2577 scope.go:117] "RemoveContainer" containerID="99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71"
May 28 23:26:05.610448 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:26:05.610432    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71\": container with ID starting with 99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71 not found: ID does not exist" containerID="99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71"
May 28 23:26:05.610489 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.610452    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71"} err="failed to get container status \"99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71\": rpc error: code = NotFound desc = could not find container \"99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71\": container with ID starting with 99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71 not found: ID does not exist"
May 28 23:26:05.610489 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.610466    2577 scope.go:117] "RemoveContainer" containerID="5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06"
May 28 23:26:05.610653 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:26:05.610639    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06\": container with ID starting with 5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06 not found: ID does not exist" containerID="5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06"
May 28 23:26:05.610701 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.610656    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06"} err="failed to get container status \"5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06\": rpc error: code = NotFound desc = could not find container \"5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06\": container with ID starting with 5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06 not found: ID does not exist"
May 28 23:26:05.610701 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.610667    2577 scope.go:117] "RemoveContainer" containerID="175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210"
May 28 23:26:05.610929 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.610909    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210"} err="failed to get container status \"175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210\": rpc error: code = NotFound desc = could not find container \"175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210\": container with ID starting with 175d1bb9f7c130f3447a8d06da81baf5ea4a4fd9e316134ba3a317c68d4a5210 not found: ID does not exist"
May 28 23:26:05.610982 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.610930    2577 scope.go:117] "RemoveContainer" containerID="cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095"
May 28 23:26:05.611119 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.611096    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095"} err="failed to get container status \"cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095\": rpc error: code = NotFound desc = could not find container \"cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095\": container with ID starting with cf3c68c919af74b64716f6404461c86398e8eeef25ede324fee5f68a29f78095 not found: ID does not exist"
May 28 23:26:05.611119 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.611111    2577 scope.go:117] "RemoveContainer" containerID="b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312"
May 28 23:26:05.611302 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.611286    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312"} err="failed to get container status \"b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312\": rpc error: code = NotFound desc = could not find container \"b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312\": container with ID starting with b8ec2edf6cebfd9a9881f98c7ea3070f82d23da85126d2999540f9f296a4f312 not found: ID does not exist"
May 28 23:26:05.611353 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.611303    2577 scope.go:117] "RemoveContainer" containerID="b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c"
May 28 23:26:05.611509 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.611494    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c"} err="failed to get container status \"b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c\": rpc error: code = NotFound desc = could not find container \"b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c\": container with ID starting with b891d96d2673e13f987a5e4868a0988a1431b5fcea638f48ffe1a41f3061244c not found: ID does not exist"
May 28 23:26:05.611558 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.611509    2577 scope.go:117] "RemoveContainer" containerID="99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71"
May 28 23:26:05.611683 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.611667    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71"} err="failed to get container status \"99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71\": rpc error: code = NotFound desc = could not find container \"99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71\": container with ID starting with 99c17cead8e9d4343da46e1422e69bc740800ffad3d301709685fae21fa7fc71 not found: ID does not exist"
May 28 23:26:05.611726 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.611684    2577 scope.go:117] "RemoveContainer" containerID="5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06"
May 28 23:26:05.611850 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:05.611836    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06"} err="failed to get container status \"5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06\": rpc error: code = NotFound desc = could not find container \"5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06\": container with ID starting with 5565d0cf9c439c252569b2ed72af2610ddca96ba00c53cf60065a0bdedeecd06 not found: ID does not exist"
May 28 23:26:07.491860 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:07.491825    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" path="/var/lib/kubelet/pods/e2715e5d-f90a-46be-883b-05d36da15d82/volumes"
May 28 23:26:56.821474 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.821436    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"]
May 28 23:26:56.822086 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.821941    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="registry"
May 28 23:26:56.822086 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.821957    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="registry"
May 28 23:26:56.822086 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.821981    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="feast-apply"
May 28 23:26:56.822086 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.821990    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="feast-apply"
May 28 23:26:56.822086 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.822009    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="feast-init"
May 28 23:26:56.822086 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.822018    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="feast-init"
May 28 23:26:56.822086 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.822027    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="online"
May 28 23:26:56.822086 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.822035    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="online"
May 28 23:26:56.822086 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.822049    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="ui"
May 28 23:26:56.822086 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.822057    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="ui"
May 28 23:26:56.822086 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.822069    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="offline"
May 28 23:26:56.822086 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.822077    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="offline"
May 28 23:26:56.822577 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.822159    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="registry"
May 28 23:26:56.822577 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.822172    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="online"
May 28 23:26:56.822577 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.822185    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="ui"
May 28 23:26:56.822577 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.822195    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="e2715e5d-f90a-46be-883b-05d36da15d82" containerName="offline"
May 28 23:26:56.826186 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.826166    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:56.828835 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.828812    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 28 23:26:56.829027 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.828989    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 28 23:26:56.829140 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.829057    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 28 23:26:56.829140 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.828916    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 28 23:26:56.829266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.828916    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 28 23:26:56.829266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.828956    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 28 23:26:56.829266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.828907    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-tzbps\""
May 28 23:26:56.836348 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.836325    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"]
May 28 23:26:56.943629 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.943592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:56.943629 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.943631    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzrz\" (UniqueName: \"kubernetes.io/projected/16303784-6dec-41f6-b452-cd34d3d7e21e-kube-api-access-4wzrz\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:56.943850 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.943651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/16303784-6dec-41f6-b452-cd34d3d7e21e-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:56.943850 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.943699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:56.943850 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.943723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/16303784-6dec-41f6-b452-cd34d3d7e21e-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:56.943850 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:56.943814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:57.045260 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:57.045216    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/16303784-6dec-41f6-b452-cd34d3d7e21e-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:57.045439 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:57.045282    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:57.045439 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:57.045366    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:57.045439 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:26:57.045376    2577 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found
May 28 23:26:57.045439 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:57.045393    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzrz\" (UniqueName: \"kubernetes.io/projected/16303784-6dec-41f6-b452-cd34d3d7e21e-kube-api-access-4wzrz\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:57.045439 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:26:57.045433    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-offline-tls podName:16303784-6dec-41f6-b452-cd34d3d7e21e nodeName:}" failed. No retries permitted until 2026-05-28 23:26:57.54541714 +0000 UTC m=+1490.738848375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-offline-tls") pod "feast-simple-feast-remote-setup-5cff94746b-tmttc" (UID: "16303784-6dec-41f6-b452-cd34d3d7e21e") : secret "feast-simple-feast-remote-setup-offline-tls" not found
May 28 23:26:57.045708 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:57.045458    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/16303784-6dec-41f6-b452-cd34d3d7e21e-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:57.045708 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:57.045500    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:57.045708 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:57.045676    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/16303784-6dec-41f6-b452-cd34d3d7e21e-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:57.046247 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:57.046211    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/16303784-6dec-41f6-b452-cd34d3d7e21e-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:57.048035 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:57.048016    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:57.048154 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:57.048135    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:57.055050 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:57.055013    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzrz\" (UniqueName: \"kubernetes.io/projected/16303784-6dec-41f6-b452-cd34d3d7e21e-kube-api-access-4wzrz\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:57.550402 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:57.550363    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:57.552845 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:57.552821    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-tmttc\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:57.738783 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:57.738749    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:26:57.872975 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:57.872791    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"]
May 28 23:26:57.875620 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:26:57.875594    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16303784_6dec_41f6_b452_cd34d3d7e21e.slice/crio-70da5d0ae0323079a7b549b9645d3d29d80213031be58dfff3ae536e46a9f366 WatchSource:0}: Error finding container 70da5d0ae0323079a7b549b9645d3d29d80213031be58dfff3ae536e46a9f366: Status 404 returned error can't find the container with id 70da5d0ae0323079a7b549b9645d3d29d80213031be58dfff3ae536e46a9f366
May 28 23:26:58.740202 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:58.740165    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" event={"ID":"16303784-6dec-41f6-b452-cd34d3d7e21e","Type":"ContainerStarted","Data":"2d250786ac40ca2f4717d6b1287ab8ab2e0a17355b5538888dfe1cda20ddacf8"}
May 28 23:26:58.740396 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:26:58.740208    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" event={"ID":"16303784-6dec-41f6-b452-cd34d3d7e21e","Type":"ContainerStarted","Data":"70da5d0ae0323079a7b549b9645d3d29d80213031be58dfff3ae536e46a9f366"}
May 28 23:27:02.754644 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:02.754610    2577 generic.go:358] "Generic (PLEG): container finished" podID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerID="2d250786ac40ca2f4717d6b1287ab8ab2e0a17355b5538888dfe1cda20ddacf8" exitCode=0
May 28 23:27:02.755075 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:02.754683    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" event={"ID":"16303784-6dec-41f6-b452-cd34d3d7e21e","Type":"ContainerDied","Data":"2d250786ac40ca2f4717d6b1287ab8ab2e0a17355b5538888dfe1cda20ddacf8"}
May 28 23:27:03.759734 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:03.759697    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" event={"ID":"16303784-6dec-41f6-b452-cd34d3d7e21e","Type":"ContainerStarted","Data":"a61ca62295531cd9cc8f66eef8a2b7ca4458f00cbd240d8609d85a04f17d7e30"}
May 28 23:27:08.778932 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:08.778895    2577 generic.go:358] "Generic (PLEG): container finished" podID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerID="a61ca62295531cd9cc8f66eef8a2b7ca4458f00cbd240d8609d85a04f17d7e30" exitCode=0
May 28 23:27:08.779318 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:08.778901    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" event={"ID":"16303784-6dec-41f6-b452-cd34d3d7e21e","Type":"ContainerDied","Data":"a61ca62295531cd9cc8f66eef8a2b7ca4458f00cbd240d8609d85a04f17d7e30"}
May 28 23:27:09.784925 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:09.784888    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" event={"ID":"16303784-6dec-41f6-b452-cd34d3d7e21e","Type":"ContainerStarted","Data":"077202ae9e81aef7e81369399a85293a09852078b032a4d740c9501bba9a3218"}
May 28 23:27:09.785405 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:09.784934    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" event={"ID":"16303784-6dec-41f6-b452-cd34d3d7e21e","Type":"ContainerStarted","Data":"fb27beb2425fc53b271ac245ad8afcc3720d31a10ea7bf84d32c105f13ef677d"}
May 28 23:27:09.785405 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:09.784948    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" event={"ID":"16303784-6dec-41f6-b452-cd34d3d7e21e","Type":"ContainerStarted","Data":"eac2386c074bb626f2a1a3d21195f03639234915aa6e1fab9a0d4718d51cee83"}
May 28 23:27:09.809238 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:09.809179    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" podStartSLOduration=13.809160293 podStartE2EDuration="13.809160293s" podCreationTimestamp="2026-05-28 23:26:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:27:09.806801006 +0000 UTC m=+1503.000232265" watchObservedRunningTime="2026-05-28 23:27:09.809160293 +0000 UTC m=+1503.002591547"
May 28 23:27:12.739259 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:12.739206    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:27:12.739259 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:12.739268    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:27:12.739827 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:12.739281    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:27:12.740968 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:12.740841    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.42:8816: connect: connection refused"
May 28 23:27:12.740968 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:12.740889    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.42:8443: connect: connection refused"
May 28 23:27:12.740968 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:12.740917    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.42:6567/health\": dial tcp 10.133.0.42:6567: connect: connection refused"
May 28 23:27:13.051932 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:13.051805    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:27:13.056312 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:13.056291    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:27:15.740138 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:15.740107    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:27:15.740608 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:15.740230    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:27:15.740729 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:15.740619    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:27:15.740729 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:15.740656    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:27:15.741039 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:15.741013    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:27:15.741162 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:15.741092    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:27:15.744407 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:15.744386    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:27:15.807317 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:15.807273    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:27:15.810297 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:27:15.810274    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:28:18.102907 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:18.102848    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"]
May 28 23:28:18.103424 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:18.103291    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="online" containerID="cri-o://eac2386c074bb626f2a1a3d21195f03639234915aa6e1fab9a0d4718d51cee83" gracePeriod=30
May 28 23:28:18.103424 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:18.103340    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="offline" containerID="cri-o://fb27beb2425fc53b271ac245ad8afcc3720d31a10ea7bf84d32c105f13ef677d" gracePeriod=30
May 28 23:28:18.103573 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:18.103340    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="ui" containerID="cri-o://077202ae9e81aef7e81369399a85293a09852078b032a4d740c9501bba9a3218" gracePeriod=30
May 28 23:28:18.247593 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:18.247557    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"]
May 28 23:28:18.248079 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:18.247920    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="registry" containerID="cri-o://c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478" gracePeriod=30
May 28 23:28:18.248079 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:18.247976    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="ui" containerID="cri-o://9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e" gracePeriod=30
May 28 23:28:18.248079 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:18.248004    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="online" containerID="cri-o://df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35" gracePeriod=30
May 28 23:28:18.248079 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:18.248004    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="offline" containerID="cri-o://639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc" gracePeriod=30
May 28 23:28:19.030486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:19.030455    2577 generic.go:358] "Generic (PLEG): container finished" podID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerID="9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e" exitCode=0
May 28 23:28:19.030486 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:19.030482    2577 generic.go:358] "Generic (PLEG): container finished" podID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerID="df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35" exitCode=0
May 28 23:28:19.030695 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:19.030539    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" event={"ID":"027df7a5-98d8-47a2-8959-80ac6dbba4ec","Type":"ContainerDied","Data":"9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e"}
May 28 23:28:19.030695 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:19.030579    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" event={"ID":"027df7a5-98d8-47a2-8959-80ac6dbba4ec","Type":"ContainerDied","Data":"df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35"}
May 28 23:28:19.032882 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:19.032838    2577 generic.go:358] "Generic (PLEG): container finished" podID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerID="077202ae9e81aef7e81369399a85293a09852078b032a4d740c9501bba9a3218" exitCode=0
May 28 23:28:19.032882 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:19.032858    2577 generic.go:358] "Generic (PLEG): container finished" podID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerID="eac2386c074bb626f2a1a3d21195f03639234915aa6e1fab9a0d4718d51cee83" exitCode=0
May 28 23:28:19.033027 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:19.032899    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" event={"ID":"16303784-6dec-41f6-b452-cd34d3d7e21e","Type":"ContainerDied","Data":"077202ae9e81aef7e81369399a85293a09852078b032a4d740c9501bba9a3218"}
May 28 23:28:19.033027 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:19.032927    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" event={"ID":"16303784-6dec-41f6-b452-cd34d3d7e21e","Type":"ContainerDied","Data":"eac2386c074bb626f2a1a3d21195f03639234915aa6e1fab9a0d4718d51cee83"}
May 28 23:28:23.852188 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:23.852139    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused"
May 28 23:28:24.524368 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:24.524318    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused"
May 28 23:28:25.741531 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:25.741486    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.42:8443: connect: connection refused"
May 28 23:28:25.807970 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:25.807929    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.42:6567/health\": dial tcp 10.133.0.42:6567: connect: connection refused"
May 28 23:28:33.852138 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:33.852091    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused"
May 28 23:28:34.524393 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:34.524343    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused"
May 28 23:28:35.741487 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:35.741437    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.42:8443: connect: connection refused"
May 28 23:28:35.807662 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:35.807621    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.42:6567/health\": dial tcp 10.133.0.42:6567: connect: connection refused"
May 28 23:28:43.852943 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:43.852861    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused"
May 28 23:28:43.853365 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:43.853054    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:28:44.524399 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:44.524355    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused"
May 28 23:28:44.524591 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:44.524470    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:28:45.740991 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:45.740944    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.42:8443: connect: connection refused"
May 28 23:28:45.741409 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:45.741067    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:28:45.808180 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:45.808140    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.42:6567/health\": dial tcp 10.133.0.42:6567: connect: connection refused"
May 28 23:28:45.808350 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:45.808249    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:28:48.349152 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.349122    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:28:48.436306 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.436222    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/16303784-6dec-41f6-b452-cd34d3d7e21e-feast-data\") pod \"16303784-6dec-41f6-b452-cd34d3d7e21e\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") "
May 28 23:28:48.436306 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.436260    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/16303784-6dec-41f6-b452-cd34d3d7e21e-registry-tls\") pod \"16303784-6dec-41f6-b452-cd34d3d7e21e\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") "
May 28 23:28:48.436306 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.436292    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-ui-tls\") pod \"16303784-6dec-41f6-b452-cd34d3d7e21e\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") "
May 28 23:28:48.436582 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.436314    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-offline-tls\") pod \"16303784-6dec-41f6-b452-cd34d3d7e21e\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") "
May 28 23:28:48.436582 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.436394    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-online-tls\") pod \"16303784-6dec-41f6-b452-cd34d3d7e21e\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") "
May 28 23:28:48.436582 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.436419    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wzrz\" (UniqueName: \"kubernetes.io/projected/16303784-6dec-41f6-b452-cd34d3d7e21e-kube-api-access-4wzrz\") pod \"16303784-6dec-41f6-b452-cd34d3d7e21e\" (UID: \"16303784-6dec-41f6-b452-cd34d3d7e21e\") "
May 28 23:28:48.436754 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.436655    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16303784-6dec-41f6-b452-cd34d3d7e21e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "16303784-6dec-41f6-b452-cd34d3d7e21e" (UID: "16303784-6dec-41f6-b452-cd34d3d7e21e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:28:48.436811 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.436753    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16303784-6dec-41f6-b452-cd34d3d7e21e-feast-data" (OuterVolumeSpecName: "feast-data") pod "16303784-6dec-41f6-b452-cd34d3d7e21e" (UID: "16303784-6dec-41f6-b452-cd34d3d7e21e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:28:48.438683 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.438652    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "16303784-6dec-41f6-b452-cd34d3d7e21e" (UID: "16303784-6dec-41f6-b452-cd34d3d7e21e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:28:48.438814 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.438686    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "16303784-6dec-41f6-b452-cd34d3d7e21e" (UID: "16303784-6dec-41f6-b452-cd34d3d7e21e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:28:48.438814 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.438701    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-online-tls" (OuterVolumeSpecName: "online-tls") pod "16303784-6dec-41f6-b452-cd34d3d7e21e" (UID: "16303784-6dec-41f6-b452-cd34d3d7e21e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:28:48.438814 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.438758    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16303784-6dec-41f6-b452-cd34d3d7e21e-kube-api-access-4wzrz" (OuterVolumeSpecName: "kube-api-access-4wzrz") pod "16303784-6dec-41f6-b452-cd34d3d7e21e" (UID: "16303784-6dec-41f6-b452-cd34d3d7e21e"). InnerVolumeSpecName "kube-api-access-4wzrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:28:48.537421 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.537389    2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-online-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:28:48.537421 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.537415    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wzrz\" (UniqueName: \"kubernetes.io/projected/16303784-6dec-41f6-b452-cd34d3d7e21e-kube-api-access-4wzrz\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:28:48.537421 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.537426    2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/16303784-6dec-41f6-b452-cd34d3d7e21e-feast-data\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:28:48.537630 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.537435    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/16303784-6dec-41f6-b452-cd34d3d7e21e-registry-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:28:48.537630 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.537443    2577 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-ui-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:28:48.537630 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.537450    2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/16303784-6dec-41f6-b452-cd34d3d7e21e-offline-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:28:48.891905 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:48.891878    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:28:49.040643 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.040554    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/027df7a5-98d8-47a2-8959-80ac6dbba4ec-feast-data\") pod \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") "
May 28 23:28:49.040643 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.040633    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-online-tls\") pod \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") "
May 28 23:28:49.040933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.040653    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-offline-tls\") pod \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") "
May 28 23:28:49.040933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.040692    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-registry-tls\") pod \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") "
May 28 23:28:49.040933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.040716    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-ui-tls\") pod \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") "
May 28 23:28:49.040933 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.040736    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bsrw\" (UniqueName: \"kubernetes.io/projected/027df7a5-98d8-47a2-8959-80ac6dbba4ec-kube-api-access-5bsrw\") pod \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\" (UID: \"027df7a5-98d8-47a2-8959-80ac6dbba4ec\") "
May 28 23:28:49.041198 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.041165    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027df7a5-98d8-47a2-8959-80ac6dbba4ec-feast-data" (OuterVolumeSpecName: "feast-data") pod "027df7a5-98d8-47a2-8959-80ac6dbba4ec" (UID: "027df7a5-98d8-47a2-8959-80ac6dbba4ec"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:28:49.043070 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.043034    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-online-tls" (OuterVolumeSpecName: "online-tls") pod "027df7a5-98d8-47a2-8959-80ac6dbba4ec" (UID: "027df7a5-98d8-47a2-8959-80ac6dbba4ec"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:28:49.043070 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.043062    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "027df7a5-98d8-47a2-8959-80ac6dbba4ec" (UID: "027df7a5-98d8-47a2-8959-80ac6dbba4ec"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:28:49.043240 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.043079    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027df7a5-98d8-47a2-8959-80ac6dbba4ec-kube-api-access-5bsrw" (OuterVolumeSpecName: "kube-api-access-5bsrw") pod "027df7a5-98d8-47a2-8959-80ac6dbba4ec" (UID: "027df7a5-98d8-47a2-8959-80ac6dbba4ec"). InnerVolumeSpecName "kube-api-access-5bsrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:28:49.043448 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.043421    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "027df7a5-98d8-47a2-8959-80ac6dbba4ec" (UID: "027df7a5-98d8-47a2-8959-80ac6dbba4ec"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:28:49.043448 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.043430    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "027df7a5-98d8-47a2-8959-80ac6dbba4ec" (UID: "027df7a5-98d8-47a2-8959-80ac6dbba4ec"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:28:49.135232 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.135204    2577 generic.go:358] "Generic (PLEG): container finished" podID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerID="fb27beb2425fc53b271ac245ad8afcc3720d31a10ea7bf84d32c105f13ef677d" exitCode=137
May 28 23:28:49.135377 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.135286    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"
May 28 23:28:49.135377 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.135291    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" event={"ID":"16303784-6dec-41f6-b452-cd34d3d7e21e","Type":"ContainerDied","Data":"fb27beb2425fc53b271ac245ad8afcc3720d31a10ea7bf84d32c105f13ef677d"}
May 28 23:28:49.135377 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.135336    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc" event={"ID":"16303784-6dec-41f6-b452-cd34d3d7e21e","Type":"ContainerDied","Data":"70da5d0ae0323079a7b549b9645d3d29d80213031be58dfff3ae536e46a9f366"}
May 28 23:28:49.135377 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.135353    2577 scope.go:117] "RemoveContainer" containerID="077202ae9e81aef7e81369399a85293a09852078b032a4d740c9501bba9a3218"
May 28 23:28:49.138097 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.138059    2577 generic.go:358] "Generic (PLEG): container finished" podID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerID="639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc" exitCode=137
May 28 23:28:49.138097 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.138085    2577 generic.go:358] "Generic (PLEG): container finished" podID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerID="c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478" exitCode=137
May 28 23:28:49.138259 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.138132    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" event={"ID":"027df7a5-98d8-47a2-8959-80ac6dbba4ec","Type":"ContainerDied","Data":"639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc"}
May 28 23:28:49.138259 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.138178    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" event={"ID":"027df7a5-98d8-47a2-8959-80ac6dbba4ec","Type":"ContainerDied","Data":"c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478"}
May 28 23:28:49.138259 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.138196    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt" event={"ID":"027df7a5-98d8-47a2-8959-80ac6dbba4ec","Type":"ContainerDied","Data":"2857e4d8fc9c59a7f4d2eb10a8b05e716d6d371120b47f6241f590312fff8768"}
May 28 23:28:49.138259 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.138145    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"
May 28 23:28:49.141365 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.141342    2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/027df7a5-98d8-47a2-8959-80ac6dbba4ec-feast-data\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:28:49.141460 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.141371    2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-online-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:28:49.141460 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.141384    2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-offline-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:28:49.141460 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.141395    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-registry-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:28:49.141460 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.141407    2577 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/027df7a5-98d8-47a2-8959-80ac6dbba4ec-ui-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:28:49.141460 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.141420    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bsrw\" (UniqueName: \"kubernetes.io/projected/027df7a5-98d8-47a2-8959-80ac6dbba4ec-kube-api-access-5bsrw\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:28:49.147290 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.147265    2577 scope.go:117] "RemoveContainer" containerID="fb27beb2425fc53b271ac245ad8afcc3720d31a10ea7bf84d32c105f13ef677d"
May 28 23:28:49.155604 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.155583    2577 scope.go:117] "RemoveContainer" containerID="eac2386c074bb626f2a1a3d21195f03639234915aa6e1fab9a0d4718d51cee83"
May 28 23:28:49.160413 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.160389    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"]
May 28 23:28:49.163753 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.163728    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-tmttc"]
May 28 23:28:49.165227 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.165210    2577 scope.go:117] "RemoveContainer" containerID="a61ca62295531cd9cc8f66eef8a2b7ca4458f00cbd240d8609d85a04f17d7e30"
May 28 23:28:49.172998 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.172978    2577 scope.go:117] "RemoveContainer" containerID="2d250786ac40ca2f4717d6b1287ab8ab2e0a17355b5538888dfe1cda20ddacf8"
May 28 23:28:49.174268 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.174247    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"]
May 28 23:28:49.180300 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.180275    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-fxgkt"]
May 28 23:28:49.181411 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.181391    2577 scope.go:117] "RemoveContainer" containerID="077202ae9e81aef7e81369399a85293a09852078b032a4d740c9501bba9a3218"
May 28 23:28:49.181671 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:28:49.181650    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077202ae9e81aef7e81369399a85293a09852078b032a4d740c9501bba9a3218\": container with ID starting with 077202ae9e81aef7e81369399a85293a09852078b032a4d740c9501bba9a3218 not found: ID does not exist" containerID="077202ae9e81aef7e81369399a85293a09852078b032a4d740c9501bba9a3218"
May 28 23:28:49.181728 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.181679    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077202ae9e81aef7e81369399a85293a09852078b032a4d740c9501bba9a3218"} err="failed to get container status \"077202ae9e81aef7e81369399a85293a09852078b032a4d740c9501bba9a3218\": rpc error: code = NotFound desc = could not find container \"077202ae9e81aef7e81369399a85293a09852078b032a4d740c9501bba9a3218\": container with ID starting with 077202ae9e81aef7e81369399a85293a09852078b032a4d740c9501bba9a3218 not found: ID does not exist"
May 28 23:28:49.181728 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.181698    2577 scope.go:117] "RemoveContainer" containerID="fb27beb2425fc53b271ac245ad8afcc3720d31a10ea7bf84d32c105f13ef677d"
May 28 23:28:49.181981 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:28:49.181965    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb27beb2425fc53b271ac245ad8afcc3720d31a10ea7bf84d32c105f13ef677d\": container with ID starting with fb27beb2425fc53b271ac245ad8afcc3720d31a10ea7bf84d32c105f13ef677d not found: ID does not exist" containerID="fb27beb2425fc53b271ac245ad8afcc3720d31a10ea7bf84d32c105f13ef677d"
May 28 23:28:49.182031 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.181986    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb27beb2425fc53b271ac245ad8afcc3720d31a10ea7bf84d32c105f13ef677d"} err="failed to get container status \"fb27beb2425fc53b271ac245ad8afcc3720d31a10ea7bf84d32c105f13ef677d\": rpc error: code = NotFound desc = could not find container \"fb27beb2425fc53b271ac245ad8afcc3720d31a10ea7bf84d32c105f13ef677d\": container with ID starting with fb27beb2425fc53b271ac245ad8afcc3720d31a10ea7bf84d32c105f13ef677d not found: ID does not exist"
May 28 23:28:49.182031 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.182000    2577 scope.go:117] "RemoveContainer" containerID="eac2386c074bb626f2a1a3d21195f03639234915aa6e1fab9a0d4718d51cee83"
May 28 23:28:49.182249 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:28:49.182233    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eac2386c074bb626f2a1a3d21195f03639234915aa6e1fab9a0d4718d51cee83\": container with ID starting with eac2386c074bb626f2a1a3d21195f03639234915aa6e1fab9a0d4718d51cee83 not found: ID does not exist" containerID="eac2386c074bb626f2a1a3d21195f03639234915aa6e1fab9a0d4718d51cee83"
May 28 23:28:49.182293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.182252    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eac2386c074bb626f2a1a3d21195f03639234915aa6e1fab9a0d4718d51cee83"} err="failed to get container status \"eac2386c074bb626f2a1a3d21195f03639234915aa6e1fab9a0d4718d51cee83\": rpc error: code = NotFound desc = could not find container \"eac2386c074bb626f2a1a3d21195f03639234915aa6e1fab9a0d4718d51cee83\": container with ID starting with eac2386c074bb626f2a1a3d21195f03639234915aa6e1fab9a0d4718d51cee83 not found: ID does not exist"
May 28 23:28:49.182293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.182263    2577 scope.go:117] "RemoveContainer" containerID="a61ca62295531cd9cc8f66eef8a2b7ca4458f00cbd240d8609d85a04f17d7e30"
May 28 23:28:49.182474 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:28:49.182458    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61ca62295531cd9cc8f66eef8a2b7ca4458f00cbd240d8609d85a04f17d7e30\": container with ID starting with a61ca62295531cd9cc8f66eef8a2b7ca4458f00cbd240d8609d85a04f17d7e30 not found: ID does not exist" containerID="a61ca62295531cd9cc8f66eef8a2b7ca4458f00cbd240d8609d85a04f17d7e30"
May 28 23:28:49.182556 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.182477    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61ca62295531cd9cc8f66eef8a2b7ca4458f00cbd240d8609d85a04f17d7e30"} err="failed to get container status \"a61ca62295531cd9cc8f66eef8a2b7ca4458f00cbd240d8609d85a04f17d7e30\": rpc error: code = NotFound desc = could not find container \"a61ca62295531cd9cc8f66eef8a2b7ca4458f00cbd240d8609d85a04f17d7e30\": container with ID starting with a61ca62295531cd9cc8f66eef8a2b7ca4458f00cbd240d8609d85a04f17d7e30 not found: ID does not exist"
May 28 23:28:49.182556 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.182489    2577 scope.go:117] "RemoveContainer" containerID="2d250786ac40ca2f4717d6b1287ab8ab2e0a17355b5538888dfe1cda20ddacf8"
May 28 23:28:49.182700 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:28:49.182683    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d250786ac40ca2f4717d6b1287ab8ab2e0a17355b5538888dfe1cda20ddacf8\": container with ID starting with 2d250786ac40ca2f4717d6b1287ab8ab2e0a17355b5538888dfe1cda20ddacf8 not found: ID does not exist" containerID="2d250786ac40ca2f4717d6b1287ab8ab2e0a17355b5538888dfe1cda20ddacf8"
May 28 23:28:49.182738 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.182705    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d250786ac40ca2f4717d6b1287ab8ab2e0a17355b5538888dfe1cda20ddacf8"} err="failed to get container status \"2d250786ac40ca2f4717d6b1287ab8ab2e0a17355b5538888dfe1cda20ddacf8\": rpc error: code = NotFound desc = could not find container \"2d250786ac40ca2f4717d6b1287ab8ab2e0a17355b5538888dfe1cda20ddacf8\": container with ID starting with 2d250786ac40ca2f4717d6b1287ab8ab2e0a17355b5538888dfe1cda20ddacf8 not found: ID does not exist"
May 28 23:28:49.182738 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.182721    2577 scope.go:117] "RemoveContainer" containerID="9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e"
May 28 23:28:49.190009 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.189995    2577 scope.go:117] "RemoveContainer" containerID="639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc"
May 28 23:28:49.196712 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.196698    2577 scope.go:117] "RemoveContainer" containerID="df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35"
May 28 23:28:49.203420 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.203406    2577 scope.go:117] "RemoveContainer" containerID="c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478"
May 28 23:28:49.210260 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.210245    2577 scope.go:117] "RemoveContainer" containerID="4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d"
May 28 23:28:49.217446 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.217418    2577 scope.go:117] "RemoveContainer" containerID="4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74"
May 28 23:28:49.224152 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.224135    2577 scope.go:117] "RemoveContainer" containerID="9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e"
May 28 23:28:49.224386 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:28:49.224366    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e\": container with ID starting with 9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e not found: ID does not exist" containerID="9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e"
May 28 23:28:49.224457 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.224396    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e"} err="failed to get container status \"9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e\": rpc error: code = NotFound desc = could not find container \"9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e\": container with ID starting with 9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e not found: ID does not exist"
May 28 23:28:49.224457 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.224423    2577 scope.go:117] "RemoveContainer" containerID="639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc"
May 28 23:28:49.224654 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:28:49.224635    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc\": container with ID starting with 639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc not found: ID does not exist" containerID="639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc"
May 28 23:28:49.224713 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.224662    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc"} err="failed to get container status \"639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc\": rpc error: code = NotFound desc = could not find container \"639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc\": container with ID starting with 639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc not found: ID does not exist"
May 28 23:28:49.224713 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.224681    2577 scope.go:117] "RemoveContainer" containerID="df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35"
May 28 23:28:49.224930 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:28:49.224908    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35\": container with ID starting with df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35 not found: ID does not exist" containerID="df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35"
May 28 23:28:49.225015 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.224931    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35"} err="failed to get container status \"df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35\": rpc error: code = NotFound desc = could not find container \"df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35\": container with ID starting with df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35 not found: ID does not exist"
May 28 23:28:49.225015 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.224945    2577 scope.go:117] "RemoveContainer" containerID="c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478"
May 28 23:28:49.225194 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:28:49.225177    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478\": container with ID starting with c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478 not found: ID does not exist" containerID="c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478"
May 28 23:28:49.225231 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.225208    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478"} err="failed to get container status \"c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478\": rpc error: code = NotFound desc = could not find container \"c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478\": container with ID starting with c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478 not found: ID does not exist"
May 28 23:28:49.225231 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.225223    2577 scope.go:117] "RemoveContainer" containerID="4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d"
May 28 23:28:49.225439 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:28:49.225424    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d\": container with ID starting with 4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d not found: ID does not exist" containerID="4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d"
May 28 23:28:49.225480 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.225446    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d"} err="failed to get container status \"4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d\": rpc error: code = NotFound desc = could not find container \"4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d\": container with ID starting with 4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d not found: ID does not exist"
May 28 23:28:49.225480 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.225466    2577 scope.go:117] "RemoveContainer" containerID="4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74"
May 28 23:28:49.225655 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:28:49.225639    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74\": container with ID starting with 4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74 not found: ID does not exist" containerID="4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74"
May 28 23:28:49.225694 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.225657    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74"} err="failed to get container status \"4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74\": rpc error: code = NotFound desc = could not find container \"4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74\": container with ID starting with 4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74 not found: ID does not exist"
May 28 23:28:49.225694 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.225672    2577 scope.go:117] "RemoveContainer" containerID="9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e"
May 28 23:28:49.225924 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.225907    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e"} err="failed to get container status \"9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e\": rpc error: code = NotFound desc = could not find container \"9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e\": container with ID starting with 9c4b3fb9fe6640d79a3507414e0df9b54d53df3081f6434bdc75c817d759d75e not found: ID does not exist"
May 28 23:28:49.225975 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.225925    2577 scope.go:117] "RemoveContainer" containerID="639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc"
May 28 23:28:49.226153 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.226136    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc"} err="failed to get container status \"639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc\": rpc error: code = NotFound desc = could not find container \"639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc\": container with ID starting with 639144cbbdb29cf77319f38bca7d47f6cdde4a26630358bd77d78df51fa272cc not found: ID does not exist"
May 28 23:28:49.226193 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.226154    2577 scope.go:117] "RemoveContainer" containerID="df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35"
May 28 23:28:49.226363 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.226343    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35"} err="failed to get container status \"df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35\": rpc error: code = NotFound desc = could not find container \"df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35\": container with ID starting with df6a18e580aa5ab62ba67bf49e8c20df9e4b164a4c5872ace721473875e2ff35 not found: ID does not exist"
May 28 23:28:49.226411 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.226365    2577 scope.go:117] "RemoveContainer" containerID="c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478"
May 28 23:28:49.226574 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.226556    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478"} err="failed to get container status \"c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478\": rpc error: code = NotFound desc = could not find container \"c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478\": container with ID starting with c37d94d1c0fc06fefa02d7b1b7c7ebd75731ec6b779feb1672f9a0f12c83f478 not found: ID does not exist"
May 28 23:28:49.226629 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.226575    2577 scope.go:117] "RemoveContainer" containerID="4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d"
May 28 23:28:49.226812 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.226793    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d"} err="failed to get container status \"4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d\": rpc error: code = NotFound desc = could not find container \"4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d\": container with ID starting with 4f23d4a9398d661773d9082c56bba18126cea548db4e2378c9c5dc52552f6d3d not found: ID does not exist"
May 28 23:28:49.226891 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.226813    2577 scope.go:117] "RemoveContainer" containerID="4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74"
May 28 23:28:49.227073 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.227053    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74"} err="failed to get container status \"4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74\": rpc error: code = NotFound desc = could not find container \"4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74\": container with ID starting with 4ed784bae9ef7a65a397ab51dbce2c5f751f7f0d5952d4ca3fafefca79326f74 not found: ID does not exist"
May 28 23:28:49.490999 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.490975    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" path="/var/lib/kubelet/pods/027df7a5-98d8-47a2-8959-80ac6dbba4ec/volumes"
May 28 23:28:49.491584 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:28:49.491571    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" path="/var/lib/kubelet/pods/16303784-6dec-41f6-b452-cd34d3d7e21e/volumes"
May 28 23:29:28.227160 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227124    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"]
May 28 23:29:28.227646 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227632    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="feast-apply"
May 28 23:29:28.227707 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227648    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="feast-apply"
May 28 23:29:28.227707 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227664    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="online"
May 28 23:29:28.227707 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227672    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="online"
May 28 23:29:28.227707 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227683    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="online"
May 28 23:29:28.227707 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227691    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="online"
May 28 23:29:28.227707 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227704    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="ui"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227712    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="ui"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227725    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="offline"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227733    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="offline"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227741    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="feast-init"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227749    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="feast-init"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227768    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="feast-apply"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227775    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="feast-apply"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227790    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="ui"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227797    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="ui"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227807    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="registry"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227814    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="registry"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227824    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="offline"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227832    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="offline"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227842    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="feast-init"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227850    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="feast-init"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227937    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="registry"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227952    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="ui"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227964    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="offline"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227974    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="offline"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227982    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="online"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.227994    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="027df7a5-98d8-47a2-8959-80ac6dbba4ec" containerName="ui"
May 28 23:29:28.228042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.228002    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="16303784-6dec-41f6-b452-cd34d3d7e21e" containerName="online"
May 28 23:29:28.233297 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.233273    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.235776 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.235752    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 28 23:29:28.235971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.235952    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-mn8qj\""
May 28 23:29:28.236042 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.235798    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 28 23:29:28.236097 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.235838    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 23:29:28.236167 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.235892    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 28 23:29:28.236221 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.235958    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 23:29:28.236277 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.235761    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 28 23:29:28.241090 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.241064    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"]
May 28 23:29:28.375935 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.375843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.375935 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.375894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.376130 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.375949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.376130 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.375977    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/95323350-3f77-44c1-a5ff-7e15c4891bf2-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.376130 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.376007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sptrb\" (UniqueName: \"kubernetes.io/projected/95323350-3f77-44c1-a5ff-7e15c4891bf2-kube-api-access-sptrb\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.376130 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.376036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.477174 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.477130    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.477368 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.477183    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.477368 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.477251    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.477368 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.477280    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/95323350-3f77-44c1-a5ff-7e15c4891bf2-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.477368 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:29:28.477287    2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 28 23:29:28.477368 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.477319    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sptrb\" (UniqueName: \"kubernetes.io/projected/95323350-3f77-44c1-a5ff-7e15c4891bf2-kube-api-access-sptrb\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.477618 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:29:28.477380    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-registry-tls podName:95323350-3f77-44c1-a5ff-7e15c4891bf2 nodeName:}" failed. No retries permitted until 2026-05-28 23:29:28.977356456 +0000 UTC m=+1642.170787698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-registry-tls") pod "feast-simple-feast-setup-664fcfcc58-rhfpm" (UID: "95323350-3f77-44c1-a5ff-7e15c4891bf2") : secret "feast-simple-feast-setup-registry-tls" not found
May 28 23:29:28.477618 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.477406    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.477855 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.477817    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/95323350-3f77-44c1-a5ff-7e15c4891bf2-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.479988 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.479965    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.480079 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.480023    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.480126 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.480094    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.485723 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.485699    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sptrb\" (UniqueName: \"kubernetes.io/projected/95323350-3f77-44c1-a5ff-7e15c4891bf2-kube-api-access-sptrb\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.982411 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.982367    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:28.985070 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:28.985041    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-rhfpm\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:29.156962 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:29.156929    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:29.291804 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:29.291775    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"]
May 28 23:29:29.292117 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:29:29.291937    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95323350_3f77_44c1_a5ff_7e15c4891bf2.slice/crio-e43ae1d00c6c54546885408d31f0f9bf23946a3166eefd427c1296f3b9b26a18 WatchSource:0}: Error finding container e43ae1d00c6c54546885408d31f0f9bf23946a3166eefd427c1296f3b9b26a18: Status 404 returned error can't find the container with id e43ae1d00c6c54546885408d31f0f9bf23946a3166eefd427c1296f3b9b26a18
May 28 23:29:29.293529 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:29.293516    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 23:29:30.271557 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:30.271520    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" event={"ID":"95323350-3f77-44c1-a5ff-7e15c4891bf2","Type":"ContainerStarted","Data":"0291ddf85f7ff12285bff8d5f243d11896b17380f2c0b352e45c06c7be30c684"}
May 28 23:29:30.271557 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:30.271559    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" event={"ID":"95323350-3f77-44c1-a5ff-7e15c4891bf2","Type":"ContainerStarted","Data":"e43ae1d00c6c54546885408d31f0f9bf23946a3166eefd427c1296f3b9b26a18"}
May 28 23:29:34.285576 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:34.285542    2577 generic.go:358] "Generic (PLEG): container finished" podID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerID="0291ddf85f7ff12285bff8d5f243d11896b17380f2c0b352e45c06c7be30c684" exitCode=0
May 28 23:29:34.285958 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:34.285617    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" event={"ID":"95323350-3f77-44c1-a5ff-7e15c4891bf2","Type":"ContainerDied","Data":"0291ddf85f7ff12285bff8d5f243d11896b17380f2c0b352e45c06c7be30c684"}
May 28 23:29:35.290637 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:35.290601    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" event={"ID":"95323350-3f77-44c1-a5ff-7e15c4891bf2","Type":"ContainerStarted","Data":"4244f5a05c5c501fcfc63c7bb49b3c4f5102a35a674a641a71d70433930d4505"}
May 28 23:29:40.308422 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:40.308391    2577 generic.go:358] "Generic (PLEG): container finished" podID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerID="4244f5a05c5c501fcfc63c7bb49b3c4f5102a35a674a641a71d70433930d4505" exitCode=0
May 28 23:29:40.308922 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:40.308473    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" event={"ID":"95323350-3f77-44c1-a5ff-7e15c4891bf2","Type":"ContainerDied","Data":"4244f5a05c5c501fcfc63c7bb49b3c4f5102a35a674a641a71d70433930d4505"}
May 28 23:29:41.316757 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:41.316714    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" event={"ID":"95323350-3f77-44c1-a5ff-7e15c4891bf2","Type":"ContainerStarted","Data":"d6fca2f8563429667f4b42a9baf6884bf62a83c7c998d8812ce6cf26e4e7810c"}
May 28 23:29:41.317274 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:41.316764    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" event={"ID":"95323350-3f77-44c1-a5ff-7e15c4891bf2","Type":"ContainerStarted","Data":"f266170a71c1e2bd83615dfb618b0595764e47e5dbd77d61d76f1dd318aa59f4"}
May 28 23:29:41.317274 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:41.316782    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" event={"ID":"95323350-3f77-44c1-a5ff-7e15c4891bf2","Type":"ContainerStarted","Data":"44a0589ee6d902f3d6e67c05e4188d208b5d717e137321ac8ade26ab792cd0a8"}
May 28 23:29:41.317274 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:41.316795    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" event={"ID":"95323350-3f77-44c1-a5ff-7e15c4891bf2","Type":"ContainerStarted","Data":"2fc005f997d6c02e76243813d2e53434ac4ac8e1b162f93a0ef6e1954511c20a"}
May 28 23:29:41.340273 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:41.340200    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" podStartSLOduration=13.340179146 podStartE2EDuration="13.340179146s" podCreationTimestamp="2026-05-28 23:29:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:29:41.338407712 +0000 UTC m=+1654.531838965" watchObservedRunningTime="2026-05-28 23:29:41.340179146 +0000 UTC m=+1654.533610399"
May 28 23:29:44.157206 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:44.157172    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:44.157758 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:44.157727    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:44.157758 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:44.157761    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:44.157988 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:44.157774    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:44.159473 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:44.159441    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.43:8443: connect: connection refused"
May 28 23:29:44.159653 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:44.159480    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.43:6567/health\": dial tcp 10.133.0.43:6567: connect: connection refused"
May 28 23:29:44.159785 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:44.159475    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.43:6571: connect: connection refused"
May 28 23:29:44.159785 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:44.159518    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.43:8816: connect: connection refused"
May 28 23:29:47.158590 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:47.158552    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:47.159128 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:47.158715    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:47.159128 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:47.159080    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:47.159266 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:47.159174    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:47.159457 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:47.159436    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:47.159579 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:47.159466    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:47.159579 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:47.159524    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:47.159579 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:47.159570    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:47.159715 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:47.159699    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:47.163337 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:47.163318    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:47.339679 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:47.339648    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:29:47.342640 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:29:47.342621    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:30:49.229964 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.229907    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"]
May 28 23:30:49.230442 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.230383    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="registry" containerID="cri-o://2fc005f997d6c02e76243813d2e53434ac4ac8e1b162f93a0ef6e1954511c20a" gracePeriod=30
May 28 23:30:49.230575 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.230488    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="offline" containerID="cri-o://f266170a71c1e2bd83615dfb618b0595764e47e5dbd77d61d76f1dd318aa59f4" gracePeriod=30
May 28 23:30:49.230730 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.230492    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="ui" containerID="cri-o://d6fca2f8563429667f4b42a9baf6884bf62a83c7c998d8812ce6cf26e4e7810c" gracePeriod=30
May 28 23:30:49.230730 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.230488    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="online" containerID="cri-o://44a0589ee6d902f3d6e67c05e4188d208b5d717e137321ac8ade26ab792cd0a8" gracePeriod=30
May 28 23:30:49.495267 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.494356    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"]
May 28 23:30:49.499169 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.499146    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:49.501597 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.501570    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-d7r7w\""
May 28 23:30:49.508300 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.507650    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"]
May 28 23:30:49.577195 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.577145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2hgt\" (UniqueName: \"kubernetes.io/projected/7189f917-716b-4ec8-84dd-cbbe526e9393-kube-api-access-s2hgt\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:49.577522 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.577505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:49.577651 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.577636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:49.577836 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.577823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:49.577965 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.577952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7189f917-716b-4ec8-84dd-cbbe526e9393-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:49.578092 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.578078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:49.678923 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.678850    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s2hgt\" (UniqueName: \"kubernetes.io/projected/7189f917-716b-4ec8-84dd-cbbe526e9393-kube-api-access-s2hgt\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:49.679135 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.678968    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:49.679135 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.678996    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:49.679135 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.679092    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:49.679135 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.679115    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7189f917-716b-4ec8-84dd-cbbe526e9393-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:49.679343 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.679145    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:49.679343 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:30:49.679286    2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 28 23:30:49.679437 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:30:49.679358    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-ui-tls podName:7189f917-716b-4ec8-84dd-cbbe526e9393 nodeName:}" failed. No retries permitted until 2026-05-28 23:30:50.179336015 +0000 UTC m=+1723.372767263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-ui-tls") pod "feast-simple-feast-setup-664fcfcc58-4mbmd" (UID: "7189f917-716b-4ec8-84dd-cbbe526e9393") : secret "feast-simple-feast-setup-ui-tls" not found
May 28 23:30:49.679437 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:30:49.679379    2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 28 23:30:49.679539 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:30:49.679452    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-offline-tls podName:7189f917-716b-4ec8-84dd-cbbe526e9393 nodeName:}" failed. No retries permitted until 2026-05-28 23:30:50.179435133 +0000 UTC m=+1723.372866378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-offline-tls") pod "feast-simple-feast-setup-664fcfcc58-4mbmd" (UID: "7189f917-716b-4ec8-84dd-cbbe526e9393") : secret "feast-simple-feast-setup-offline-tls" not found
May 28 23:30:49.679598 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:30:49.679581    2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 28 23:30:49.679647 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:30:49.679620    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-online-tls podName:7189f917-716b-4ec8-84dd-cbbe526e9393 nodeName:}" failed. No retries permitted until 2026-05-28 23:30:50.179607469 +0000 UTC m=+1723.373038702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-online-tls") pod "feast-simple-feast-setup-664fcfcc58-4mbmd" (UID: "7189f917-716b-4ec8-84dd-cbbe526e9393") : secret "feast-simple-feast-setup-online-tls" not found
May 28 23:30:49.680052 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.680014    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7189f917-716b-4ec8-84dd-cbbe526e9393-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:49.682722 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.682694    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:49.687827 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:49.687798    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2hgt\" (UniqueName: \"kubernetes.io/projected/7189f917-716b-4ec8-84dd-cbbe526e9393-kube-api-access-s2hgt\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:50.184534 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:50.184442    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:50.184534 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:50.184499    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:50.184534 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:50.184525    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:50.187096 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:50.187071    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:50.187207 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:50.187111    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:50.187247 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:50.187211    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-4mbmd\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:50.412197 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:50.412154    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:30:50.545781 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:50.545745    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"]
May 28 23:30:50.547465 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:50.547441    2577 generic.go:358] "Generic (PLEG): container finished" podID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerID="d6fca2f8563429667f4b42a9baf6884bf62a83c7c998d8812ce6cf26e4e7810c" exitCode=0
May 28 23:30:50.547523 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:50.547464    2577 generic.go:358] "Generic (PLEG): container finished" podID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerID="44a0589ee6d902f3d6e67c05e4188d208b5d717e137321ac8ade26ab792cd0a8" exitCode=0
May 28 23:30:50.547523 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:50.547483    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" event={"ID":"95323350-3f77-44c1-a5ff-7e15c4891bf2","Type":"ContainerDied","Data":"d6fca2f8563429667f4b42a9baf6884bf62a83c7c998d8812ce6cf26e4e7810c"}
May 28 23:30:50.547523 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:50.547503    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" event={"ID":"95323350-3f77-44c1-a5ff-7e15c4891bf2","Type":"ContainerDied","Data":"44a0589ee6d902f3d6e67c05e4188d208b5d717e137321ac8ade26ab792cd0a8"}
May 28 23:30:50.549426 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:30:50.549405    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7189f917_716b_4ec8_84dd_cbbe526e9393.slice/crio-fa945c9272a4514b068a94040ed619dd574d303c07640d53a6e559f9f9f38b0d WatchSource:0}: Error finding container fa945c9272a4514b068a94040ed619dd574d303c07640d53a6e559f9f9f38b0d: Status 404 returned error can't find the container with id fa945c9272a4514b068a94040ed619dd574d303c07640d53a6e559f9f9f38b0d
May 28 23:30:51.551795 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:51.551760    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" event={"ID":"7189f917-716b-4ec8-84dd-cbbe526e9393","Type":"ContainerStarted","Data":"6c9339ae7b793b28d705927c12a8547ab3d55560befaac94b39a5cf10d384807"}
May 28 23:30:51.551795 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:51.551795    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" event={"ID":"7189f917-716b-4ec8-84dd-cbbe526e9393","Type":"ContainerStarted","Data":"fa945c9272a4514b068a94040ed619dd574d303c07640d53a6e559f9f9f38b0d"}
May 28 23:30:55.566346 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:55.566309    2577 generic.go:358] "Generic (PLEG): container finished" podID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerID="6c9339ae7b793b28d705927c12a8547ab3d55560befaac94b39a5cf10d384807" exitCode=0
May 28 23:30:55.566768 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:55.566396    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" event={"ID":"7189f917-716b-4ec8-84dd-cbbe526e9393","Type":"ContainerDied","Data":"6c9339ae7b793b28d705927c12a8547ab3d55560befaac94b39a5cf10d384807"}
May 28 23:30:56.571538 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:56.571501    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" event={"ID":"7189f917-716b-4ec8-84dd-cbbe526e9393","Type":"ContainerStarted","Data":"afa89aa95c81f3e70ba26f66d697008894d83e86ec3834a4662e207a90fab13e"}
May 28 23:30:57.159572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:57.159530    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.43:8443: connect: connection refused"
May 28 23:30:57.340360 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:30:57.340314    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.43:6567/health\": dial tcp 10.133.0.43:6567: connect: connection refused"
May 28 23:31:01.592965 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:01.592925    2577 generic.go:358] "Generic (PLEG): container finished" podID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerID="afa89aa95c81f3e70ba26f66d697008894d83e86ec3834a4662e207a90fab13e" exitCode=0
May 28 23:31:01.593449 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:01.593005    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" event={"ID":"7189f917-716b-4ec8-84dd-cbbe526e9393","Type":"ContainerDied","Data":"afa89aa95c81f3e70ba26f66d697008894d83e86ec3834a4662e207a90fab13e"}
May 28 23:31:02.601132 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:02.601092    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" event={"ID":"7189f917-716b-4ec8-84dd-cbbe526e9393","Type":"ContainerStarted","Data":"d3fd6c8f073fb32cc0ec0873a88866c78e8e920621b58b5c11e7d4cd3c4bc5a2"}
May 28 23:31:02.601132 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:02.601139    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" event={"ID":"7189f917-716b-4ec8-84dd-cbbe526e9393","Type":"ContainerStarted","Data":"cfa06c59be6e74dc3ff0c9a45f7f51a2050ba32e9673abec7e1018abaee47f6b"}
May 28 23:31:02.601701 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:02.601154    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" event={"ID":"7189f917-716b-4ec8-84dd-cbbe526e9393","Type":"ContainerStarted","Data":"f4214a381a8f9ef55259587d5e594a16242635074d06588e30be3b0d7db3c482"}
May 28 23:31:02.601701 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:02.601167    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" event={"ID":"7189f917-716b-4ec8-84dd-cbbe526e9393","Type":"ContainerStarted","Data":"bf66da939a2ca5b1ec9de9abd81fa6891d8f1b11c67b8be86b1dc02208bec37a"}
May 28 23:31:02.625497 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:02.625431    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" podStartSLOduration=13.625409354 podStartE2EDuration="13.625409354s" podCreationTimestamp="2026-05-28 23:30:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:31:02.624167771 +0000 UTC m=+1735.817599023" watchObservedRunningTime="2026-05-28 23:31:02.625409354 +0000 UTC m=+1735.818840607"
May 28 23:31:05.413409 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:05.413342    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:05.413409 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:05.413404    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:05.413409 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:05.413418    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:05.414064 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:05.413430    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:05.415596 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:05.415563    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.44:6571: connect: connection refused"
May 28 23:31:05.415825 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:05.415588    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.44:8443: connect: connection refused"
May 28 23:31:05.415948 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:05.415623    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="online" probeResult="failure" output="Get \"https://10.133.0.44:6567/health\": dial tcp 10.133.0.44:6567: connect: connection refused"
May 28 23:31:05.415948 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:05.415677    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.44:8816: connect: connection refused"
May 28 23:31:07.159963 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:07.159909    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.43:8443: connect: connection refused"
May 28 23:31:07.340493 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:07.340439    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.43:6567/health\": dial tcp 10.133.0.43:6567: connect: connection refused"
May 28 23:31:08.413714 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:08.413677    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:08.414153 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:08.413796    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:08.414153 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:08.414071    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:08.414435 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:08.414408    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:08.414576 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:08.414445    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:08.414576 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:08.414460    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:08.414685 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:08.414633    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:08.415002 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:08.414984    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:08.415074 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:08.415034    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:08.418121 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:08.418106    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:08.637356 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:08.637320    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:08.640426 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:08.640403    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:31:17.159347 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:17.159300    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.43:8443: connect: connection refused"
May 28 23:31:17.159849 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:17.159430    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:31:17.340582 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:17.340542    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.43:6567/health\": dial tcp 10.133.0.43:6567: connect: connection refused"
May 28 23:31:17.340751 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:17.340681    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:31:19.679011 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.678975    2577 generic.go:358] "Generic (PLEG): container finished" podID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerID="f266170a71c1e2bd83615dfb618b0595764e47e5dbd77d61d76f1dd318aa59f4" exitCode=137
May 28 23:31:19.679011 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.679003    2577 generic.go:358] "Generic (PLEG): container finished" podID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerID="2fc005f997d6c02e76243813d2e53434ac4ac8e1b162f93a0ef6e1954511c20a" exitCode=137
May 28 23:31:19.679408 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.679041    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" event={"ID":"95323350-3f77-44c1-a5ff-7e15c4891bf2","Type":"ContainerDied","Data":"f266170a71c1e2bd83615dfb618b0595764e47e5dbd77d61d76f1dd318aa59f4"}
May 28 23:31:19.679408 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.679073    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" event={"ID":"95323350-3f77-44c1-a5ff-7e15c4891bf2","Type":"ContainerDied","Data":"2fc005f997d6c02e76243813d2e53434ac4ac8e1b162f93a0ef6e1954511c20a"}
May 28 23:31:19.880489 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.880466    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:31:19.951226 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.951148    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-online-tls\") pod \"95323350-3f77-44c1-a5ff-7e15c4891bf2\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") "
May 28 23:31:19.951226 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.951180    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-offline-tls\") pod \"95323350-3f77-44c1-a5ff-7e15c4891bf2\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") "
May 28 23:31:19.951226 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.951214    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-registry-tls\") pod \"95323350-3f77-44c1-a5ff-7e15c4891bf2\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") "
May 28 23:31:19.951495 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.951251    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-ui-tls\") pod \"95323350-3f77-44c1-a5ff-7e15c4891bf2\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") "
May 28 23:31:19.951495 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.951294    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/95323350-3f77-44c1-a5ff-7e15c4891bf2-feast-data\") pod \"95323350-3f77-44c1-a5ff-7e15c4891bf2\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") "
May 28 23:31:19.951495 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.951335    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sptrb\" (UniqueName: \"kubernetes.io/projected/95323350-3f77-44c1-a5ff-7e15c4891bf2-kube-api-access-sptrb\") pod \"95323350-3f77-44c1-a5ff-7e15c4891bf2\" (UID: \"95323350-3f77-44c1-a5ff-7e15c4891bf2\") "
May 28 23:31:19.952103 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.951850    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95323350-3f77-44c1-a5ff-7e15c4891bf2-feast-data" (OuterVolumeSpecName: "feast-data") pod "95323350-3f77-44c1-a5ff-7e15c4891bf2" (UID: "95323350-3f77-44c1-a5ff-7e15c4891bf2"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:31:19.953614 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.953584    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-online-tls" (OuterVolumeSpecName: "online-tls") pod "95323350-3f77-44c1-a5ff-7e15c4891bf2" (UID: "95323350-3f77-44c1-a5ff-7e15c4891bf2"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:31:19.953737 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.953648    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95323350-3f77-44c1-a5ff-7e15c4891bf2-kube-api-access-sptrb" (OuterVolumeSpecName: "kube-api-access-sptrb") pod "95323350-3f77-44c1-a5ff-7e15c4891bf2" (UID: "95323350-3f77-44c1-a5ff-7e15c4891bf2"). InnerVolumeSpecName "kube-api-access-sptrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:31:19.953737 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.953713    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "95323350-3f77-44c1-a5ff-7e15c4891bf2" (UID: "95323350-3f77-44c1-a5ff-7e15c4891bf2"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:31:19.953949 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.953927    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "95323350-3f77-44c1-a5ff-7e15c4891bf2" (UID: "95323350-3f77-44c1-a5ff-7e15c4891bf2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:31:19.954003 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:19.953955    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "95323350-3f77-44c1-a5ff-7e15c4891bf2" (UID: "95323350-3f77-44c1-a5ff-7e15c4891bf2"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:31:20.052551 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.052498    2577 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-ui-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:31:20.052551 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.052548    2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/95323350-3f77-44c1-a5ff-7e15c4891bf2-feast-data\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:31:20.052551 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.052559    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sptrb\" (UniqueName: \"kubernetes.io/projected/95323350-3f77-44c1-a5ff-7e15c4891bf2-kube-api-access-sptrb\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:31:20.052551 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.052569    2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-online-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:31:20.052830 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.052579    2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-offline-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:31:20.052830 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.052588    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/95323350-3f77-44c1-a5ff-7e15c4891bf2-registry-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:31:20.684815 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.684769    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm" event={"ID":"95323350-3f77-44c1-a5ff-7e15c4891bf2","Type":"ContainerDied","Data":"e43ae1d00c6c54546885408d31f0f9bf23946a3166eefd427c1296f3b9b26a18"}
May 28 23:31:20.684815 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.684809    2577 scope.go:117] "RemoveContainer" containerID="d6fca2f8563429667f4b42a9baf6884bf62a83c7c998d8812ce6cf26e4e7810c"
May 28 23:31:20.685275 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.684807    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"
May 28 23:31:20.693938 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.693917    2577 scope.go:117] "RemoveContainer" containerID="f266170a71c1e2bd83615dfb618b0595764e47e5dbd77d61d76f1dd318aa59f4"
May 28 23:31:20.701540 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.701523    2577 scope.go:117] "RemoveContainer" containerID="44a0589ee6d902f3d6e67c05e4188d208b5d717e137321ac8ade26ab792cd0a8"
May 28 23:31:20.707137 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.707108    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"]
May 28 23:31:20.709372 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.709354    2577 scope.go:117] "RemoveContainer" containerID="2fc005f997d6c02e76243813d2e53434ac4ac8e1b162f93a0ef6e1954511c20a"
May 28 23:31:20.712046 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.712025    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-rhfpm"]
May 28 23:31:20.716979 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.716962    2577 scope.go:117] "RemoveContainer" containerID="4244f5a05c5c501fcfc63c7bb49b3c4f5102a35a674a641a71d70433930d4505"
May 28 23:31:20.724211 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:20.724193    2577 scope.go:117] "RemoveContainer" containerID="0291ddf85f7ff12285bff8d5f243d11896b17380f2c0b352e45c06c7be30c684"
May 28 23:31:21.491625 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:31:21.491592    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" path="/var/lib/kubelet/pods/95323350-3f77-44c1-a5ff-7e15c4891bf2/volumes"
May 28 23:32:10.958849 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.958807    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"]
May 28 23:32:10.959371 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959298    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="offline"
May 28 23:32:10.959371 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959316    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="offline"
May 28 23:32:10.959371 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959338    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="ui"
May 28 23:32:10.959371 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959343    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="ui"
May 28 23:32:10.959371 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959353    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="registry"
May 28 23:32:10.959371 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959358    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="registry"
May 28 23:32:10.959371 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959367    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="online"
May 28 23:32:10.959371 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959372    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="online"
May 28 23:32:10.959649 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959384    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="feast-apply"
May 28 23:32:10.959649 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959389    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="feast-apply"
May 28 23:32:10.959649 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959397    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="feast-init"
May 28 23:32:10.959649 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959407    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="feast-init"
May 28 23:32:10.959649 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959462    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="offline"
May 28 23:32:10.959649 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959471    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="ui"
May 28 23:32:10.959649 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959481    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="registry"
May 28 23:32:10.959649 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.959493    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="95323350-3f77-44c1-a5ff-7e15c4891bf2" containerName="online"
May 28 23:32:10.962834 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.962817    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:10.965651 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.965378    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 28 23:32:10.965651 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.965411    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 28 23:32:10.965651 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.965472    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 28 23:32:10.965651 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.965491    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 28 23:32:10.965651 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.965573    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 28 23:32:10.965651 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.965623    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-h82gm\""
May 28 23:32:10.966045 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.965697    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 28 23:32:10.974670 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.974638    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"]
May 28 23:32:10.993031 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.992999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:10.993167 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.993036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1b87a27e-79c2-4201-8dca-06bd1a7878e5-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:10.993167 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.993121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/1b87a27e-79c2-4201-8dca-06bd1a7878e5-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:10.993279 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.993171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:10.993279 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.993268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:10.993386 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:10.993300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-527n5\" (UniqueName: \"kubernetes.io/projected/1b87a27e-79c2-4201-8dca-06bd1a7878e5-kube-api-access-527n5\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:11.094426 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:11.094389    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:11.094426 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:11.094429    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1b87a27e-79c2-4201-8dca-06bd1a7878e5-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:11.094676 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:11.094479    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/1b87a27e-79c2-4201-8dca-06bd1a7878e5-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:11.094676 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:11.094528    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:11.094676 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:32:11.094551    2577 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found
May 28 23:32:11.094676 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:11.094580    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:11.094676 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:32:11.094646    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-ui-tls podName:1b87a27e-79c2-4201-8dca-06bd1a7878e5 nodeName:}" failed. No retries permitted until 2026-05-28 23:32:11.594621407 +0000 UTC m=+1804.788052652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-ui-tls") pod "feast-simple-feast-remote-setup-5cff94746b-qfm4d" (UID: "1b87a27e-79c2-4201-8dca-06bd1a7878e5") : secret "feast-simple-feast-remote-setup-ui-tls" not found
May 28 23:32:11.094676 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:32:11.094660    2577 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found
May 28 23:32:11.095035 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:32:11.094722    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-offline-tls podName:1b87a27e-79c2-4201-8dca-06bd1a7878e5 nodeName:}" failed. No retries permitted until 2026-05-28 23:32:11.59470164 +0000 UTC m=+1804.788132898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-offline-tls") pod "feast-simple-feast-remote-setup-5cff94746b-qfm4d" (UID: "1b87a27e-79c2-4201-8dca-06bd1a7878e5") : secret "feast-simple-feast-remote-setup-offline-tls" not found
May 28 23:32:11.095035 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:11.094759    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-527n5\" (UniqueName: \"kubernetes.io/projected/1b87a27e-79c2-4201-8dca-06bd1a7878e5-kube-api-access-527n5\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:11.095035 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:11.094918    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1b87a27e-79c2-4201-8dca-06bd1a7878e5-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:11.095336 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:11.095316    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/1b87a27e-79c2-4201-8dca-06bd1a7878e5-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:11.097206 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:11.097183    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:11.104487 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:11.104464    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-527n5\" (UniqueName: \"kubernetes.io/projected/1b87a27e-79c2-4201-8dca-06bd1a7878e5-kube-api-access-527n5\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:11.600106 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:11.600069    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:11.600312 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:11.600132    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:11.602690 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:11.602661    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:11.602690 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:11.602673    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-qfm4d\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:11.875669 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:11.875576    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:12.008516 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:12.008459    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"]
May 28 23:32:12.011293 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:32:12.011263    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b87a27e_79c2_4201_8dca_06bd1a7878e5.slice/crio-e4830a608f3adcb469a9b3a2a1df7a200f2a5b0637fe56dc66118bd992e65ac1 WatchSource:0}: Error finding container e4830a608f3adcb469a9b3a2a1df7a200f2a5b0637fe56dc66118bd992e65ac1: Status 404 returned error can't find the container with id e4830a608f3adcb469a9b3a2a1df7a200f2a5b0637fe56dc66118bd992e65ac1
May 28 23:32:12.873380 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:12.873336    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" event={"ID":"1b87a27e-79c2-4201-8dca-06bd1a7878e5","Type":"ContainerStarted","Data":"42906e1bacab9f3281240baedd252ff5adb62293b30a4ab73b956ed16131c620"}
May 28 23:32:12.873380 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:12.873386    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" event={"ID":"1b87a27e-79c2-4201-8dca-06bd1a7878e5","Type":"ContainerStarted","Data":"e4830a608f3adcb469a9b3a2a1df7a200f2a5b0637fe56dc66118bd992e65ac1"}
May 28 23:32:13.083194 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:13.083086    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:32:13.094226 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:13.086486    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:32:16.888455 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:16.888362    2577 generic.go:358] "Generic (PLEG): container finished" podID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerID="42906e1bacab9f3281240baedd252ff5adb62293b30a4ab73b956ed16131c620" exitCode=0
May 28 23:32:16.888805 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:16.888442    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" event={"ID":"1b87a27e-79c2-4201-8dca-06bd1a7878e5","Type":"ContainerDied","Data":"42906e1bacab9f3281240baedd252ff5adb62293b30a4ab73b956ed16131c620"}
May 28 23:32:17.894155 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:17.894117    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" event={"ID":"1b87a27e-79c2-4201-8dca-06bd1a7878e5","Type":"ContainerStarted","Data":"e0b9a78622007800e07d6275d05cd7b6f0acab1b45b56d99858d2359c4ee4fc9"}
May 28 23:32:22.911703 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:22.911667    2577 generic.go:358] "Generic (PLEG): container finished" podID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerID="e0b9a78622007800e07d6275d05cd7b6f0acab1b45b56d99858d2359c4ee4fc9" exitCode=0
May 28 23:32:22.912107 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:22.911743    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" event={"ID":"1b87a27e-79c2-4201-8dca-06bd1a7878e5","Type":"ContainerDied","Data":"e0b9a78622007800e07d6275d05cd7b6f0acab1b45b56d99858d2359c4ee4fc9"}
May 28 23:32:23.918353 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:23.918315    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" event={"ID":"1b87a27e-79c2-4201-8dca-06bd1a7878e5","Type":"ContainerStarted","Data":"d4233fddcf257d4edf67972381b7c53dd4d6dac99727762401f6572975bd9b66"}
May 28 23:32:23.918769 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:23.918366    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" event={"ID":"1b87a27e-79c2-4201-8dca-06bd1a7878e5","Type":"ContainerStarted","Data":"67bc108927c1d922df6dec431cf98a0c106f590cb75e1db3c96afd36239a7c07"}
May 28 23:32:23.918769 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:23.918383    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" event={"ID":"1b87a27e-79c2-4201-8dca-06bd1a7878e5","Type":"ContainerStarted","Data":"07a22e7b7cdccfcedfd34e976956b3c677b89597fc956fa9382803be6ed9761d"}
May 28 23:32:23.943635 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:23.943578    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" podStartSLOduration=13.943558674 podStartE2EDuration="13.943558674s" podCreationTimestamp="2026-05-28 23:32:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:32:23.941706544 +0000 UTC m=+1817.135137797" watchObservedRunningTime="2026-05-28 23:32:23.943558674 +0000 UTC m=+1817.136989927"
May 28 23:32:26.876793 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:26.876754    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:26.877399 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:26.877346    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:26.877399 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:26.877384    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:26.878336 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:26.878300    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.45:8816: connect: connection refused"
May 28 23:32:26.878336 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:26.878319    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.45:8443: connect: connection refused"
May 28 23:32:26.878531 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:26.878358    2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.45:6567/health\": dial tcp 10.133.0.45:6567: connect: connection refused"
May 28 23:32:29.876708 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:29.876671    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:29.877240 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:29.877216    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:29.877315 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:29.877256    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:29.877547 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:29.877524    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:29.881245 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:29.881223    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:29.938967 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:29.938938    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:29.939131 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:29.938976    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:29.939601 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:29.939578    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:32:29.941960 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:32:29.941940    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:33:31.935736 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:31.935702    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"]
May 28 23:33:31.936210 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:31.936014    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="online" containerID="cri-o://07a22e7b7cdccfcedfd34e976956b3c677b89597fc956fa9382803be6ed9761d" gracePeriod=30
May 28 23:33:31.936210 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:31.936122    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="offline" containerID="cri-o://67bc108927c1d922df6dec431cf98a0c106f590cb75e1db3c96afd36239a7c07" gracePeriod=30
May 28 23:33:31.936210 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:31.936154    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="ui" containerID="cri-o://d4233fddcf257d4edf67972381b7c53dd4d6dac99727762401f6572975bd9b66" gracePeriod=30
May 28 23:33:32.093293 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:32.093259    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"]
May 28 23:33:32.093640 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:32.093608    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="registry" containerID="cri-o://bf66da939a2ca5b1ec9de9abd81fa6891d8f1b11c67b8be86b1dc02208bec37a" gracePeriod=30
May 28 23:33:32.093773 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:32.093642    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="ui" containerID="cri-o://d3fd6c8f073fb32cc0ec0873a88866c78e8e920621b58b5c11e7d4cd3c4bc5a2" gracePeriod=30
May 28 23:33:32.093773 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:32.093698    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="offline" containerID="cri-o://cfa06c59be6e74dc3ff0c9a45f7f51a2050ba32e9673abec7e1018abaee47f6b" gracePeriod=30
May 28 23:33:32.093773 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:32.093755    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="online" containerID="cri-o://f4214a381a8f9ef55259587d5e594a16242635074d06588e30be3b0d7db3c482" gracePeriod=30
May 28 23:33:33.154307 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:33.154219    2577 generic.go:358] "Generic (PLEG): container finished" podID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerID="d3fd6c8f073fb32cc0ec0873a88866c78e8e920621b58b5c11e7d4cd3c4bc5a2" exitCode=0
May 28 23:33:33.154307 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:33.154245    2577 generic.go:358] "Generic (PLEG): container finished" podID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerID="f4214a381a8f9ef55259587d5e594a16242635074d06588e30be3b0d7db3c482" exitCode=0
May 28 23:33:33.154307 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:33.154293    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" event={"ID":"7189f917-716b-4ec8-84dd-cbbe526e9393","Type":"ContainerDied","Data":"d3fd6c8f073fb32cc0ec0873a88866c78e8e920621b58b5c11e7d4cd3c4bc5a2"}
May 28 23:33:33.154792 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:33.154329    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" event={"ID":"7189f917-716b-4ec8-84dd-cbbe526e9393","Type":"ContainerDied","Data":"f4214a381a8f9ef55259587d5e594a16242635074d06588e30be3b0d7db3c482"}
May 28 23:33:33.156596 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:33.156570    2577 generic.go:358] "Generic (PLEG): container finished" podID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerID="d4233fddcf257d4edf67972381b7c53dd4d6dac99727762401f6572975bd9b66" exitCode=0
May 28 23:33:33.156596 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:33.156592    2577 generic.go:358] "Generic (PLEG): container finished" podID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerID="07a22e7b7cdccfcedfd34e976956b3c677b89597fc956fa9382803be6ed9761d" exitCode=0
May 28 23:33:33.156746 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:33.156646    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" event={"ID":"1b87a27e-79c2-4201-8dca-06bd1a7878e5","Type":"ContainerDied","Data":"d4233fddcf257d4edf67972381b7c53dd4d6dac99727762401f6572975bd9b66"}
May 28 23:33:33.156746 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:33.156673    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" event={"ID":"1b87a27e-79c2-4201-8dca-06bd1a7878e5","Type":"ContainerDied","Data":"07a22e7b7cdccfcedfd34e976956b3c677b89597fc956fa9382803be6ed9761d"}
May 28 23:33:38.415373 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:38.415325    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.44:8443: connect: connection refused"
May 28 23:33:38.637590 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:38.637546    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="online" probeResult="failure" output="Get \"https://10.133.0.44:6567/health\": dial tcp 10.133.0.44:6567: connect: connection refused"
May 28 23:33:39.939826 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:39.939781    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.45:8443: connect: connection refused"
May 28 23:33:39.940311 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:39.939835    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.45:6567/health\": dial tcp 10.133.0.45:6567: connect: connection refused"
May 28 23:33:48.414973 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:48.414916    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.44:8443: connect: connection refused"
May 28 23:33:48.637946 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:48.637905    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="online" probeResult="failure" output="Get \"https://10.133.0.44:6567/health\": dial tcp 10.133.0.44:6567: connect: connection refused"
May 28 23:33:49.939526 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:49.939473    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.45:8443: connect: connection refused"
May 28 23:33:49.940013 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:49.939576    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.45:6567/health\": dial tcp 10.133.0.45:6567: connect: connection refused"
May 28 23:33:58.415155 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:58.415104    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.44:8443: connect: connection refused"
May 28 23:33:58.415671 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:58.415242    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:33:58.637698 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:58.637655    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="online" probeResult="failure" output="Get \"https://10.133.0.44:6567/health\": dial tcp 10.133.0.44:6567: connect: connection refused"
May 28 23:33:58.637889 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:58.637773    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:33:59.939280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:59.939237    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.45:8443: connect: connection refused"
May 28 23:33:59.939280 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:59.939265    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.45:6567/health\": dial tcp 10.133.0.45:6567: connect: connection refused"
May 28 23:33:59.939707 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:59.939360    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:33:59.939707 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:33:59.939402    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:34:02.116686 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.116663    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:34:02.160515 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.160483    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1b87a27e-79c2-4201-8dca-06bd1a7878e5-feast-data\") pod \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") "
May 28 23:34:02.160695 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.160526    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/1b87a27e-79c2-4201-8dca-06bd1a7878e5-registry-tls\") pod \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") "
May 28 23:34:02.160695 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.160551    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-ui-tls\") pod \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") "
May 28 23:34:02.160695 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.160582    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-online-tls\") pod \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") "
May 28 23:34:02.160695 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.160608    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-527n5\" (UniqueName: \"kubernetes.io/projected/1b87a27e-79c2-4201-8dca-06bd1a7878e5-kube-api-access-527n5\") pod \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") "
May 28 23:34:02.160695 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.160639    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-offline-tls\") pod \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\" (UID: \"1b87a27e-79c2-4201-8dca-06bd1a7878e5\") "
May 28 23:34:02.160996 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.160827    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b87a27e-79c2-4201-8dca-06bd1a7878e5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1b87a27e-79c2-4201-8dca-06bd1a7878e5" (UID: "1b87a27e-79c2-4201-8dca-06bd1a7878e5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 23:34:02.160996 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.160978    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/1b87a27e-79c2-4201-8dca-06bd1a7878e5-registry-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:34:02.161103 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.161044    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b87a27e-79c2-4201-8dca-06bd1a7878e5-feast-data" (OuterVolumeSpecName: "feast-data") pod "1b87a27e-79c2-4201-8dca-06bd1a7878e5" (UID: "1b87a27e-79c2-4201-8dca-06bd1a7878e5"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:34:02.163081 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.163057    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "1b87a27e-79c2-4201-8dca-06bd1a7878e5" (UID: "1b87a27e-79c2-4201-8dca-06bd1a7878e5"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:34:02.163219 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.163076    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "1b87a27e-79c2-4201-8dca-06bd1a7878e5" (UID: "1b87a27e-79c2-4201-8dca-06bd1a7878e5"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:34:02.163337 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.163319    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b87a27e-79c2-4201-8dca-06bd1a7878e5-kube-api-access-527n5" (OuterVolumeSpecName: "kube-api-access-527n5") pod "1b87a27e-79c2-4201-8dca-06bd1a7878e5" (UID: "1b87a27e-79c2-4201-8dca-06bd1a7878e5"). InnerVolumeSpecName "kube-api-access-527n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:34:02.163472 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.163455    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-online-tls" (OuterVolumeSpecName: "online-tls") pod "1b87a27e-79c2-4201-8dca-06bd1a7878e5" (UID: "1b87a27e-79c2-4201-8dca-06bd1a7878e5"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:34:02.260490 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.260459    2577 generic.go:358] "Generic (PLEG): container finished" podID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerID="cfa06c59be6e74dc3ff0c9a45f7f51a2050ba32e9673abec7e1018abaee47f6b" exitCode=137
May 28 23:34:02.260490 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.260489    2577 generic.go:358] "Generic (PLEG): container finished" podID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerID="bf66da939a2ca5b1ec9de9abd81fa6891d8f1b11c67b8be86b1dc02208bec37a" exitCode=137
May 28 23:34:02.260695 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.260536    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" event={"ID":"7189f917-716b-4ec8-84dd-cbbe526e9393","Type":"ContainerDied","Data":"cfa06c59be6e74dc3ff0c9a45f7f51a2050ba32e9673abec7e1018abaee47f6b"}
May 28 23:34:02.260695 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.260571    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" event={"ID":"7189f917-716b-4ec8-84dd-cbbe526e9393","Type":"ContainerDied","Data":"bf66da939a2ca5b1ec9de9abd81fa6891d8f1b11c67b8be86b1dc02208bec37a"}
May 28 23:34:02.261409 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.261388    2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1b87a27e-79c2-4201-8dca-06bd1a7878e5-feast-data\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:34:02.261508 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.261414    2577 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-ui-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:34:02.261508 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.261428    2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-online-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:34:02.261508 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.261443    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-527n5\" (UniqueName: \"kubernetes.io/projected/1b87a27e-79c2-4201-8dca-06bd1a7878e5-kube-api-access-527n5\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:34:02.261508 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.261459    2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1b87a27e-79c2-4201-8dca-06bd1a7878e5-offline-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:34:02.262724 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.262698    2577 generic.go:358] "Generic (PLEG): container finished" podID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerID="67bc108927c1d922df6dec431cf98a0c106f590cb75e1db3c96afd36239a7c07" exitCode=137
May 28 23:34:02.262836 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.262786    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"
May 28 23:34:02.262836 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.262787    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" event={"ID":"1b87a27e-79c2-4201-8dca-06bd1a7878e5","Type":"ContainerDied","Data":"67bc108927c1d922df6dec431cf98a0c106f590cb75e1db3c96afd36239a7c07"}
May 28 23:34:02.262836 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.262820    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d" event={"ID":"1b87a27e-79c2-4201-8dca-06bd1a7878e5","Type":"ContainerDied","Data":"e4830a608f3adcb469a9b3a2a1df7a200f2a5b0637fe56dc66118bd992e65ac1"}
May 28 23:34:02.262836 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.262836    2577 scope.go:117] "RemoveContainer" containerID="d4233fddcf257d4edf67972381b7c53dd4d6dac99727762401f6572975bd9b66"
May 28 23:34:02.272043 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.272029    2577 scope.go:117] "RemoveContainer" containerID="67bc108927c1d922df6dec431cf98a0c106f590cb75e1db3c96afd36239a7c07"
May 28 23:34:02.284604 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.284579    2577 scope.go:117] "RemoveContainer" containerID="07a22e7b7cdccfcedfd34e976956b3c677b89597fc956fa9382803be6ed9761d"
May 28 23:34:02.285384 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.285359    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"]
May 28 23:34:02.291184 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.291159    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-qfm4d"]
May 28 23:34:02.292501 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.292488    2577 scope.go:117] "RemoveContainer" containerID="e0b9a78622007800e07d6275d05cd7b6f0acab1b45b56d99858d2359c4ee4fc9"
May 28 23:34:02.299999 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.299983    2577 scope.go:117] "RemoveContainer" containerID="42906e1bacab9f3281240baedd252ff5adb62293b30a4ab73b956ed16131c620"
May 28 23:34:02.306963 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.306944    2577 scope.go:117] "RemoveContainer" containerID="d4233fddcf257d4edf67972381b7c53dd4d6dac99727762401f6572975bd9b66"
May 28 23:34:02.307212 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:34:02.307190    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4233fddcf257d4edf67972381b7c53dd4d6dac99727762401f6572975bd9b66\": container with ID starting with d4233fddcf257d4edf67972381b7c53dd4d6dac99727762401f6572975bd9b66 not found: ID does not exist" containerID="d4233fddcf257d4edf67972381b7c53dd4d6dac99727762401f6572975bd9b66"
May 28 23:34:02.307290 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.307218    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4233fddcf257d4edf67972381b7c53dd4d6dac99727762401f6572975bd9b66"} err="failed to get container status \"d4233fddcf257d4edf67972381b7c53dd4d6dac99727762401f6572975bd9b66\": rpc error: code = NotFound desc = could not find container \"d4233fddcf257d4edf67972381b7c53dd4d6dac99727762401f6572975bd9b66\": container with ID starting with d4233fddcf257d4edf67972381b7c53dd4d6dac99727762401f6572975bd9b66 not found: ID does not exist"
May 28 23:34:02.307290 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.307238    2577 scope.go:117] "RemoveContainer" containerID="67bc108927c1d922df6dec431cf98a0c106f590cb75e1db3c96afd36239a7c07"
May 28 23:34:02.307447 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:34:02.307431    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67bc108927c1d922df6dec431cf98a0c106f590cb75e1db3c96afd36239a7c07\": container with ID starting with 67bc108927c1d922df6dec431cf98a0c106f590cb75e1db3c96afd36239a7c07 not found: ID does not exist" containerID="67bc108927c1d922df6dec431cf98a0c106f590cb75e1db3c96afd36239a7c07"
May 28 23:34:02.307487 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.307454    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67bc108927c1d922df6dec431cf98a0c106f590cb75e1db3c96afd36239a7c07"} err="failed to get container status \"67bc108927c1d922df6dec431cf98a0c106f590cb75e1db3c96afd36239a7c07\": rpc error: code = NotFound desc = could not find container \"67bc108927c1d922df6dec431cf98a0c106f590cb75e1db3c96afd36239a7c07\": container with ID starting with 67bc108927c1d922df6dec431cf98a0c106f590cb75e1db3c96afd36239a7c07 not found: ID does not exist"
May 28 23:34:02.307487 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.307469    2577 scope.go:117] "RemoveContainer" containerID="07a22e7b7cdccfcedfd34e976956b3c677b89597fc956fa9382803be6ed9761d"
May 28 23:34:02.307701 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:34:02.307682    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a22e7b7cdccfcedfd34e976956b3c677b89597fc956fa9382803be6ed9761d\": container with ID starting with 07a22e7b7cdccfcedfd34e976956b3c677b89597fc956fa9382803be6ed9761d not found: ID does not exist" containerID="07a22e7b7cdccfcedfd34e976956b3c677b89597fc956fa9382803be6ed9761d"
May 28 23:34:02.307750 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.307703    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a22e7b7cdccfcedfd34e976956b3c677b89597fc956fa9382803be6ed9761d"} err="failed to get container status \"07a22e7b7cdccfcedfd34e976956b3c677b89597fc956fa9382803be6ed9761d\": rpc error: code = NotFound desc = could not find container \"07a22e7b7cdccfcedfd34e976956b3c677b89597fc956fa9382803be6ed9761d\": container with ID starting with 07a22e7b7cdccfcedfd34e976956b3c677b89597fc956fa9382803be6ed9761d not found: ID does not exist"
May 28 23:34:02.307750 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.307715    2577 scope.go:117] "RemoveContainer" containerID="e0b9a78622007800e07d6275d05cd7b6f0acab1b45b56d99858d2359c4ee4fc9"
May 28 23:34:02.307945 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:34:02.307926    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b9a78622007800e07d6275d05cd7b6f0acab1b45b56d99858d2359c4ee4fc9\": container with ID starting with e0b9a78622007800e07d6275d05cd7b6f0acab1b45b56d99858d2359c4ee4fc9 not found: ID does not exist" containerID="e0b9a78622007800e07d6275d05cd7b6f0acab1b45b56d99858d2359c4ee4fc9"
May 28 23:34:02.308039 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.307948    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b9a78622007800e07d6275d05cd7b6f0acab1b45b56d99858d2359c4ee4fc9"} err="failed to get container status \"e0b9a78622007800e07d6275d05cd7b6f0acab1b45b56d99858d2359c4ee4fc9\": rpc error: code = NotFound desc = could not find container \"e0b9a78622007800e07d6275d05cd7b6f0acab1b45b56d99858d2359c4ee4fc9\": container with ID starting with e0b9a78622007800e07d6275d05cd7b6f0acab1b45b56d99858d2359c4ee4fc9 not found: ID does not exist"
May 28 23:34:02.308039 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.307960    2577 scope.go:117] "RemoveContainer" containerID="42906e1bacab9f3281240baedd252ff5adb62293b30a4ab73b956ed16131c620"
May 28 23:34:02.308197 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:34:02.308182    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42906e1bacab9f3281240baedd252ff5adb62293b30a4ab73b956ed16131c620\": container with ID starting with 42906e1bacab9f3281240baedd252ff5adb62293b30a4ab73b956ed16131c620 not found: ID does not exist" containerID="42906e1bacab9f3281240baedd252ff5adb62293b30a4ab73b956ed16131c620"
May 28 23:34:02.308237 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.308199    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42906e1bacab9f3281240baedd252ff5adb62293b30a4ab73b956ed16131c620"} err="failed to get container status \"42906e1bacab9f3281240baedd252ff5adb62293b30a4ab73b956ed16131c620\": rpc error: code = NotFound desc = could not find container \"42906e1bacab9f3281240baedd252ff5adb62293b30a4ab73b956ed16131c620\": container with ID starting with 42906e1bacab9f3281240baedd252ff5adb62293b30a4ab73b956ed16131c620 not found: ID does not exist"
May 28 23:34:02.742437 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.742413    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:34:02.868493 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.868393    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-offline-tls\") pod \"7189f917-716b-4ec8-84dd-cbbe526e9393\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") "
May 28 23:34:02.868493 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.868462    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2hgt\" (UniqueName: \"kubernetes.io/projected/7189f917-716b-4ec8-84dd-cbbe526e9393-kube-api-access-s2hgt\") pod \"7189f917-716b-4ec8-84dd-cbbe526e9393\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") "
May 28 23:34:02.868741 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.868511    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7189f917-716b-4ec8-84dd-cbbe526e9393-feast-data\") pod \"7189f917-716b-4ec8-84dd-cbbe526e9393\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") "
May 28 23:34:02.868741 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.868528    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-ui-tls\") pod \"7189f917-716b-4ec8-84dd-cbbe526e9393\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") "
May 28 23:34:02.868741 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.868543    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-online-tls\") pod \"7189f917-716b-4ec8-84dd-cbbe526e9393\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") "
May 28 23:34:02.868741 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.868564    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-registry-tls\") pod \"7189f917-716b-4ec8-84dd-cbbe526e9393\" (UID: \"7189f917-716b-4ec8-84dd-cbbe526e9393\") "
May 28 23:34:02.869102 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.869074    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7189f917-716b-4ec8-84dd-cbbe526e9393-feast-data" (OuterVolumeSpecName: "feast-data") pod "7189f917-716b-4ec8-84dd-cbbe526e9393" (UID: "7189f917-716b-4ec8-84dd-cbbe526e9393"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:34:02.870839 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.870801    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-online-tls" (OuterVolumeSpecName: "online-tls") pod "7189f917-716b-4ec8-84dd-cbbe526e9393" (UID: "7189f917-716b-4ec8-84dd-cbbe526e9393"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:34:02.870839 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.870815    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "7189f917-716b-4ec8-84dd-cbbe526e9393" (UID: "7189f917-716b-4ec8-84dd-cbbe526e9393"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:34:02.871025 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.870885    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7189f917-716b-4ec8-84dd-cbbe526e9393" (UID: "7189f917-716b-4ec8-84dd-cbbe526e9393"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:34:02.871155 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.871138    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7189f917-716b-4ec8-84dd-cbbe526e9393-kube-api-access-s2hgt" (OuterVolumeSpecName: "kube-api-access-s2hgt") pod "7189f917-716b-4ec8-84dd-cbbe526e9393" (UID: "7189f917-716b-4ec8-84dd-cbbe526e9393"). InnerVolumeSpecName "kube-api-access-s2hgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:34:02.871225 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.871207    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "7189f917-716b-4ec8-84dd-cbbe526e9393" (UID: "7189f917-716b-4ec8-84dd-cbbe526e9393"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 23:34:02.970102 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.970075    2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-offline-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:34:02.970205 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.970104    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2hgt\" (UniqueName: \"kubernetes.io/projected/7189f917-716b-4ec8-84dd-cbbe526e9393-kube-api-access-s2hgt\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:34:02.970205 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.970114    2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/7189f917-716b-4ec8-84dd-cbbe526e9393-feast-data\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:34:02.970205 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.970124    2577 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-ui-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:34:02.970205 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.970132    2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-online-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:34:02.970205 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:02.970140    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/7189f917-716b-4ec8-84dd-cbbe526e9393-registry-tls\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:34:03.268809 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:03.268783    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"
May 28 23:34:03.269282 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:03.268771    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd" event={"ID":"7189f917-716b-4ec8-84dd-cbbe526e9393","Type":"ContainerDied","Data":"fa945c9272a4514b068a94040ed619dd574d303c07640d53a6e559f9f9f38b0d"}
May 28 23:34:03.269282 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:03.268916    2577 scope.go:117] "RemoveContainer" containerID="d3fd6c8f073fb32cc0ec0873a88866c78e8e920621b58b5c11e7d4cd3c4bc5a2"
May 28 23:34:03.277599 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:03.277578    2577 scope.go:117] "RemoveContainer" containerID="cfa06c59be6e74dc3ff0c9a45f7f51a2050ba32e9673abec7e1018abaee47f6b"
May 28 23:34:03.286090 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:03.286037    2577 scope.go:117] "RemoveContainer" containerID="f4214a381a8f9ef55259587d5e594a16242635074d06588e30be3b0d7db3c482"
May 28 23:34:03.291964 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:03.291903    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"]
May 28 23:34:03.294418 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:03.294338    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-4mbmd"]
May 28 23:34:03.297667 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:03.297642    2577 scope.go:117] "RemoveContainer" containerID="bf66da939a2ca5b1ec9de9abd81fa6891d8f1b11c67b8be86b1dc02208bec37a"
May 28 23:34:03.304930 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:03.304913    2577 scope.go:117] "RemoveContainer" containerID="afa89aa95c81f3e70ba26f66d697008894d83e86ec3834a4662e207a90fab13e"
May 28 23:34:03.312011 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:03.311981    2577 scope.go:117] "RemoveContainer" containerID="6c9339ae7b793b28d705927c12a8547ab3d55560befaac94b39a5cf10d384807"
May 28 23:34:03.491328 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:03.491292    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" path="/var/lib/kubelet/pods/1b87a27e-79c2-4201-8dca-06bd1a7878e5/volumes"
May 28 23:34:03.491833 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:03.491820    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" path="/var/lib/kubelet/pods/7189f917-716b-4ec8-84dd-cbbe526e9393/volumes"
May 28 23:34:21.046519 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046481    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kzqtg/must-gather-gk7jc"]
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046833    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="feast-apply"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046848    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="feast-apply"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046884    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="ui"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046892    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="ui"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046905    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="feast-init"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046911    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="feast-init"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046918    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="online"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046922    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="online"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046930    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="offline"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046935    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="offline"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046942    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="feast-apply"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046947    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="feast-apply"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046953    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="online"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046958    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="online"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046965    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="ui"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046969    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="ui"
May 28 23:34:21.046971 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046977    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="offline"
May 28 23:34:21.047519 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046982    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="offline"
May 28 23:34:21.047519 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046991    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="registry"
May 28 23:34:21.047519 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.046996    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="registry"
May 28 23:34:21.047519 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.047001    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="feast-init"
May 28 23:34:21.047519 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.047006    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="feast-init"
May 28 23:34:21.047519 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.047061    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="ui"
May 28 23:34:21.047519 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.047070    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="offline"
May 28 23:34:21.047519 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.047077    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="offline"
May 28 23:34:21.047519 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.047083    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="1b87a27e-79c2-4201-8dca-06bd1a7878e5" containerName="online"
May 28 23:34:21.047519 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.047089    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="registry"
May 28 23:34:21.047519 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.047096    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="online"
May 28 23:34:21.047519 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.047102    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="7189f917-716b-4ec8-84dd-cbbe526e9393" containerName="ui"
May 28 23:34:21.051504 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.051479    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kzqtg/must-gather-gk7jc"
May 28 23:34:21.054347 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.054327    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kzqtg\"/\"openshift-service-ca.crt\""
May 28 23:34:21.054482 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.054367    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kzqtg\"/\"default-dockercfg-pzjqx\""
May 28 23:34:21.055472 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.055447    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kzqtg\"/\"kube-root-ca.crt\""
May 28 23:34:21.057479 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.057456    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kzqtg/must-gather-gk7jc"]
May 28 23:34:21.238257 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.238214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d4rl\" (UniqueName: \"kubernetes.io/projected/a636442b-48de-4608-8aa7-f61eaf4cc8e8-kube-api-access-4d4rl\") pod \"must-gather-gk7jc\" (UID: \"a636442b-48de-4608-8aa7-f61eaf4cc8e8\") " pod="openshift-must-gather-kzqtg/must-gather-gk7jc"
May 28 23:34:21.238443 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.238278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a636442b-48de-4608-8aa7-f61eaf4cc8e8-must-gather-output\") pod \"must-gather-gk7jc\" (UID: \"a636442b-48de-4608-8aa7-f61eaf4cc8e8\") " pod="openshift-must-gather-kzqtg/must-gather-gk7jc"
May 28 23:34:21.339611 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.339519    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4d4rl\" (UniqueName: \"kubernetes.io/projected/a636442b-48de-4608-8aa7-f61eaf4cc8e8-kube-api-access-4d4rl\") pod \"must-gather-gk7jc\" (UID: \"a636442b-48de-4608-8aa7-f61eaf4cc8e8\") " pod="openshift-must-gather-kzqtg/must-gather-gk7jc"
May 28 23:34:21.339611 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.339578    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a636442b-48de-4608-8aa7-f61eaf4cc8e8-must-gather-output\") pod \"must-gather-gk7jc\" (UID: \"a636442b-48de-4608-8aa7-f61eaf4cc8e8\") " pod="openshift-must-gather-kzqtg/must-gather-gk7jc"
May 28 23:34:21.339900 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.339855    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a636442b-48de-4608-8aa7-f61eaf4cc8e8-must-gather-output\") pod \"must-gather-gk7jc\" (UID: \"a636442b-48de-4608-8aa7-f61eaf4cc8e8\") " pod="openshift-must-gather-kzqtg/must-gather-gk7jc"
May 28 23:34:21.348572 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.348538    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d4rl\" (UniqueName: \"kubernetes.io/projected/a636442b-48de-4608-8aa7-f61eaf4cc8e8-kube-api-access-4d4rl\") pod \"must-gather-gk7jc\" (UID: \"a636442b-48de-4608-8aa7-f61eaf4cc8e8\") " pod="openshift-must-gather-kzqtg/must-gather-gk7jc"
May 28 23:34:21.370045 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.370019    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kzqtg/must-gather-gk7jc"
May 28 23:34:21.498058 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:21.498035    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kzqtg/must-gather-gk7jc"]
May 28 23:34:21.500833 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:34:21.500804    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda636442b_48de_4608_8aa7_f61eaf4cc8e8.slice/crio-ff2b7b601e65b956acb7b2687a880bf1e561ed597a4c652f32edbdcb09f4a441 WatchSource:0}: Error finding container ff2b7b601e65b956acb7b2687a880bf1e561ed597a4c652f32edbdcb09f4a441: Status 404 returned error can't find the container with id ff2b7b601e65b956acb7b2687a880bf1e561ed597a4c652f32edbdcb09f4a441
May 28 23:34:22.332583 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:22.332541    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kzqtg/must-gather-gk7jc" event={"ID":"a636442b-48de-4608-8aa7-f61eaf4cc8e8","Type":"ContainerStarted","Data":"ff2b7b601e65b956acb7b2687a880bf1e561ed597a4c652f32edbdcb09f4a441"}
May 28 23:34:28.355210 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:28.355175    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kzqtg/must-gather-gk7jc" event={"ID":"a636442b-48de-4608-8aa7-f61eaf4cc8e8","Type":"ContainerStarted","Data":"e1f455dfcbf3218affff1a3a61bf17476afefb245d062ac4db5595ebf89a1fd7"}
May 28 23:34:28.355616 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:28.355217    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kzqtg/must-gather-gk7jc" event={"ID":"a636442b-48de-4608-8aa7-f61eaf4cc8e8","Type":"ContainerStarted","Data":"419ad2fa377f8078385eb2db4035a376a2622cfda23cda041609b5d67e569919"}
May 28 23:34:28.373202 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:28.373153    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-kzqtg/must-gather-gk7jc" podStartSLOduration=1.268446097 podStartE2EDuration="7.373139347s" podCreationTimestamp="2026-05-28 23:34:21 +0000 UTC" firstStartedPulling="2026-05-28 23:34:21.502901069 +0000 UTC m=+1934.696332302" lastFinishedPulling="2026-05-28 23:34:27.607594322 +0000 UTC m=+1940.801025552" observedRunningTime="2026-05-28 23:34:28.371953959 +0000 UTC m=+1941.565385212" watchObservedRunningTime="2026-05-28 23:34:28.373139347 +0000 UTC m=+1941.566570636"
May 28 23:34:37.383620 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:37.383590    2577 generic.go:358] "Generic (PLEG): container finished" podID="a636442b-48de-4608-8aa7-f61eaf4cc8e8" containerID="419ad2fa377f8078385eb2db4035a376a2622cfda23cda041609b5d67e569919" exitCode=0
May 28 23:34:37.384077 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:37.383630    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kzqtg/must-gather-gk7jc" event={"ID":"a636442b-48de-4608-8aa7-f61eaf4cc8e8","Type":"ContainerDied","Data":"419ad2fa377f8078385eb2db4035a376a2622cfda23cda041609b5d67e569919"}
May 28 23:34:37.384077 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:37.383947    2577 scope.go:117] "RemoveContainer" containerID="419ad2fa377f8078385eb2db4035a376a2622cfda23cda041609b5d67e569919"
May 28 23:34:38.316235 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:38.316201    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kzqtg_must-gather-gk7jc_a636442b-48de-4608-8aa7-f61eaf4cc8e8/gather/0.log"
May 28 23:34:41.539920 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:41.539851    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xtwjw_1f610866-a2a5-4395-bcad-928e420c46bb/global-pull-secret-syncer/0.log"
May 28 23:34:41.624936 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:41.624895    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hs8lt_9ad0103a-d3c7-4e1a-953d-1555182e1170/konnectivity-agent/0.log"
May 28 23:34:41.711425 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:41.711393    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-104.ec2.internal_ed64232379ece85cb86969c0ef54ec63/haproxy/0.log"
May 28 23:34:43.638949 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:43.638910    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-kzqtg/must-gather-gk7jc"]
May 28 23:34:43.639341 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:43.639121    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-kzqtg/must-gather-gk7jc" podUID="a636442b-48de-4608-8aa7-f61eaf4cc8e8" containerName="copy" containerID="cri-o://e1f455dfcbf3218affff1a3a61bf17476afefb245d062ac4db5595ebf89a1fd7" gracePeriod=2
May 28 23:34:43.645652 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:43.645620    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-kzqtg/must-gather-gk7jc"]
May 28 23:34:43.870361 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:43.870338    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kzqtg_must-gather-gk7jc_a636442b-48de-4608-8aa7-f61eaf4cc8e8/copy/0.log"
May 28 23:34:43.870717 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:43.870698    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kzqtg/must-gather-gk7jc"
May 28 23:34:43.872691 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:43.872663    2577 status_manager.go:895] "Failed to get status for pod" podUID="a636442b-48de-4608-8aa7-f61eaf4cc8e8" pod="openshift-must-gather-kzqtg/must-gather-gk7jc" err="pods \"must-gather-gk7jc\" is forbidden: User \"system:node:ip-10-0-143-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kzqtg\": no relationship found between node 'ip-10-0-143-104.ec2.internal' and this object"
May 28 23:34:43.931166 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:43.931075    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d4rl\" (UniqueName: \"kubernetes.io/projected/a636442b-48de-4608-8aa7-f61eaf4cc8e8-kube-api-access-4d4rl\") pod \"a636442b-48de-4608-8aa7-f61eaf4cc8e8\" (UID: \"a636442b-48de-4608-8aa7-f61eaf4cc8e8\") "
May 28 23:34:43.931166 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:43.931147    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a636442b-48de-4608-8aa7-f61eaf4cc8e8-must-gather-output\") pod \"a636442b-48de-4608-8aa7-f61eaf4cc8e8\" (UID: \"a636442b-48de-4608-8aa7-f61eaf4cc8e8\") "
May 28 23:34:43.931553 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:43.931524    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a636442b-48de-4608-8aa7-f61eaf4cc8e8-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "a636442b-48de-4608-8aa7-f61eaf4cc8e8" (UID: "a636442b-48de-4608-8aa7-f61eaf4cc8e8"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 23:34:43.933490 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:43.933467    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a636442b-48de-4608-8aa7-f61eaf4cc8e8-kube-api-access-4d4rl" (OuterVolumeSpecName: "kube-api-access-4d4rl") pod "a636442b-48de-4608-8aa7-f61eaf4cc8e8" (UID: "a636442b-48de-4608-8aa7-f61eaf4cc8e8"). InnerVolumeSpecName "kube-api-access-4d4rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 23:34:44.032763 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.032722    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d4rl\" (UniqueName: \"kubernetes.io/projected/a636442b-48de-4608-8aa7-f61eaf4cc8e8-kube-api-access-4d4rl\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:34:44.032763 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.032752    2577 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a636442b-48de-4608-8aa7-f61eaf4cc8e8-must-gather-output\") on node \"ip-10-0-143-104.ec2.internal\" DevicePath \"\""
May 28 23:34:44.332477 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.332445    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5537b0f-d674-4098-9fa9-7f3221703da8/alertmanager/0.log"
May 28 23:34:44.357278 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.357246    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5537b0f-d674-4098-9fa9-7f3221703da8/config-reloader/0.log"
May 28 23:34:44.387962 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.387935    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5537b0f-d674-4098-9fa9-7f3221703da8/kube-rbac-proxy-web/0.log"
May 28 23:34:44.406454 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.406432    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kzqtg_must-gather-gk7jc_a636442b-48de-4608-8aa7-f61eaf4cc8e8/copy/0.log"
May 28 23:34:44.406792 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.406770    2577 generic.go:358] "Generic (PLEG): container finished" podID="a636442b-48de-4608-8aa7-f61eaf4cc8e8" containerID="e1f455dfcbf3218affff1a3a61bf17476afefb245d062ac4db5595ebf89a1fd7" exitCode=143
May 28 23:34:44.406894 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.406818    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kzqtg/must-gather-gk7jc"
May 28 23:34:44.406894 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.406846    2577 scope.go:117] "RemoveContainer" containerID="e1f455dfcbf3218affff1a3a61bf17476afefb245d062ac4db5595ebf89a1fd7"
May 28 23:34:44.409099 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.409070    2577 status_manager.go:895] "Failed to get status for pod" podUID="a636442b-48de-4608-8aa7-f61eaf4cc8e8" pod="openshift-must-gather-kzqtg/must-gather-gk7jc" err="pods \"must-gather-gk7jc\" is forbidden: User \"system:node:ip-10-0-143-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kzqtg\": no relationship found between node 'ip-10-0-143-104.ec2.internal' and this object"
May 28 23:34:44.414757 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.414738    2577 scope.go:117] "RemoveContainer" containerID="419ad2fa377f8078385eb2db4035a376a2622cfda23cda041609b5d67e569919"
May 28 23:34:44.418138 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.418110    2577 status_manager.go:895] "Failed to get status for pod" podUID="a636442b-48de-4608-8aa7-f61eaf4cc8e8" pod="openshift-must-gather-kzqtg/must-gather-gk7jc" err="pods \"must-gather-gk7jc\" is forbidden: User \"system:node:ip-10-0-143-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kzqtg\": no relationship found between node 'ip-10-0-143-104.ec2.internal' and this object"
May 28 23:34:44.421012 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.420991    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5537b0f-d674-4098-9fa9-7f3221703da8/kube-rbac-proxy/0.log"
May 28 23:34:44.425509 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.425493    2577 scope.go:117] "RemoveContainer" containerID="e1f455dfcbf3218affff1a3a61bf17476afefb245d062ac4db5595ebf89a1fd7"
May 28 23:34:44.425767 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:34:44.425747    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f455dfcbf3218affff1a3a61bf17476afefb245d062ac4db5595ebf89a1fd7\": container with ID starting with e1f455dfcbf3218affff1a3a61bf17476afefb245d062ac4db5595ebf89a1fd7 not found: ID does not exist" containerID="e1f455dfcbf3218affff1a3a61bf17476afefb245d062ac4db5595ebf89a1fd7"
May 28 23:34:44.425816 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.425778    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f455dfcbf3218affff1a3a61bf17476afefb245d062ac4db5595ebf89a1fd7"} err="failed to get container status \"e1f455dfcbf3218affff1a3a61bf17476afefb245d062ac4db5595ebf89a1fd7\": rpc error: code = NotFound desc = could not find container \"e1f455dfcbf3218affff1a3a61bf17476afefb245d062ac4db5595ebf89a1fd7\": container with ID starting with e1f455dfcbf3218affff1a3a61bf17476afefb245d062ac4db5595ebf89a1fd7 not found: ID does not exist"
May 28 23:34:44.425816 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.425794    2577 scope.go:117] "RemoveContainer" containerID="419ad2fa377f8078385eb2db4035a376a2622cfda23cda041609b5d67e569919"
May 28 23:34:44.426040 ip-10-0-143-104 kubenswrapper[2577]: E0528 23:34:44.426023    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"419ad2fa377f8078385eb2db4035a376a2622cfda23cda041609b5d67e569919\": container with ID starting with 419ad2fa377f8078385eb2db4035a376a2622cfda23cda041609b5d67e569919 not found: ID does not exist" containerID="419ad2fa377f8078385eb2db4035a376a2622cfda23cda041609b5d67e569919"
May 28 23:34:44.426082 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.426046    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"419ad2fa377f8078385eb2db4035a376a2622cfda23cda041609b5d67e569919"} err="failed to get container status \"419ad2fa377f8078385eb2db4035a376a2622cfda23cda041609b5d67e569919\": rpc error: code = NotFound desc = could not find container \"419ad2fa377f8078385eb2db4035a376a2622cfda23cda041609b5d67e569919\": container with ID starting with 419ad2fa377f8078385eb2db4035a376a2622cfda23cda041609b5d67e569919 not found: ID does not exist"
May 28 23:34:44.448682 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.448664    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5537b0f-d674-4098-9fa9-7f3221703da8/kube-rbac-proxy-metric/0.log"
May 28 23:34:44.476185 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.476163    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5537b0f-d674-4098-9fa9-7f3221703da8/prom-label-proxy/0.log"
May 28 23:34:44.501182 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.501151    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5537b0f-d674-4098-9fa9-7f3221703da8/init-config-reloader/0.log"
May 28 23:34:44.539172 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.539146    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-xk7gh_9d86f162-8349-42e1-abd1-aef9e7c328a4/cluster-monitoring-operator/0.log"
May 28 23:34:44.579917 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.579893    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-bmlpg_4e3666ea-79b7-428b-bb95-96ff6ba5febc/kube-state-metrics/0.log"
May 28 23:34:44.599545 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.599469    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-bmlpg_4e3666ea-79b7-428b-bb95-96ff6ba5febc/kube-rbac-proxy-main/0.log"
May 28 23:34:44.619013 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.618986    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-bmlpg_4e3666ea-79b7-428b-bb95-96ff6ba5febc/kube-rbac-proxy-self/0.log"
May 28 23:34:44.649437 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.649399    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5bc944fdd7-djthh_f5528f72-6719-4181-b465-d1f8d01e2431/metrics-server/0.log"
May 28 23:34:44.848310 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.848274    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ns559_4a8c8899-8fd2-4dfa-a219-41a0da90e36b/node-exporter/0.log"
May 28 23:34:44.868199 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.868131    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ns559_4a8c8899-8fd2-4dfa-a219-41a0da90e36b/kube-rbac-proxy/0.log"
May 28 23:34:44.887509 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.887489    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ns559_4a8c8899-8fd2-4dfa-a219-41a0da90e36b/init-textfile/0.log"
May 28 23:34:44.915401 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.915378    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-w76gr_c848eb78-984e-41f9-becd-7cdb443b1f51/kube-rbac-proxy-main/0.log"
May 28 23:34:44.937561 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.937536    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-w76gr_c848eb78-984e-41f9-becd-7cdb443b1f51/kube-rbac-proxy-self/0.log"
May 28 23:34:44.959169 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:44.959142    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-w76gr_c848eb78-984e-41f9-becd-7cdb443b1f51/openshift-state-metrics/0.log"
May 28 23:34:45.174450 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:45.174370    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-z8lmq_b54001fb-0ec0-4dc2-8e17-8138b6ae0484/prometheus-operator-admission-webhook/0.log"
May 28 23:34:45.203043 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:45.203020    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b97578f6d-kmc5w_a45ac38f-34d6-495f-9774-5627bb58bcd6/telemeter-client/0.log"
May 28 23:34:45.221032 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:45.221005    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b97578f6d-kmc5w_a45ac38f-34d6-495f-9774-5627bb58bcd6/reload/0.log"
May 28 23:34:45.240855 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:45.240835    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b97578f6d-kmc5w_a45ac38f-34d6-495f-9774-5627bb58bcd6/kube-rbac-proxy/0.log"
May 28 23:34:45.491751 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:45.491722    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a636442b-48de-4608-8aa7-f61eaf4cc8e8" path="/var/lib/kubelet/pods/a636442b-48de-4608-8aa7-f61eaf4cc8e8/volumes"
May 28 23:34:46.456269 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:46.456237    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-d8mbh_b710a64b-729e-41cc-ab6a-fef55d47205d/networking-console-plugin/0.log"
May 28 23:34:46.850759 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:46.850724    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/2.log"
May 28 23:34:46.856004 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:46.855985    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-nj29z_edcd544f-4680-4190-a9b0-453053515b7a/console-operator/3.log"
May 28 23:34:47.187965 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:47.187887    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c6874b6d6-wvdxl_94352028-175a-4d33-836e-f99b7397441a/console/0.log"
May 28 23:34:47.222853 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:47.222821    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-6xh5k_94ca47b1-b8a4-47f6-b3ac-b4f806bf1d0e/download-server/0.log"
May 28 23:34:47.558525 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:47.558480    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-zzfbm_e9d3fa19-567d-4f02-bd2c-1dd014be825e/volume-data-source-validator/0.log"
May 28 23:34:47.949061 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:47.948978    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"]
May 28 23:34:47.949321 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:47.949308    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a636442b-48de-4608-8aa7-f61eaf4cc8e8" containerName="gather"
May 28 23:34:47.949365 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:47.949323    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a636442b-48de-4608-8aa7-f61eaf4cc8e8" containerName="gather"
May 28 23:34:47.949365 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:47.949335    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a636442b-48de-4608-8aa7-f61eaf4cc8e8" containerName="copy"
May 28 23:34:47.949365 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:47.949340    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a636442b-48de-4608-8aa7-f61eaf4cc8e8" containerName="copy"
May 28 23:34:47.949457 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:47.949403    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="a636442b-48de-4608-8aa7-f61eaf4cc8e8" containerName="copy"
May 28 23:34:47.949457 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:47.949416    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="a636442b-48de-4608-8aa7-f61eaf4cc8e8" containerName="gather"
May 28 23:34:47.952772 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:47.952751    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:47.955109 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:47.955085    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vppqb\"/\"kube-root-ca.crt\""
May 28 23:34:47.955222 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:47.955086    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vppqb\"/\"openshift-service-ca.crt\""
May 28 23:34:47.955222 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:47.955138    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vppqb\"/\"default-dockercfg-pj9km\""
May 28 23:34:47.961057 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:47.960998    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"]
May 28 23:34:48.070298 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.070261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce309e35-7eb2-43de-b885-71bc7b74e26e-sys\") pod \"perf-node-gather-daemonset-7qvrf\" (UID: \"ce309e35-7eb2-43de-b885-71bc7b74e26e\") " pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.070298 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.070300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46mld\" (UniqueName: \"kubernetes.io/projected/ce309e35-7eb2-43de-b885-71bc7b74e26e-kube-api-access-46mld\") pod \"perf-node-gather-daemonset-7qvrf\" (UID: \"ce309e35-7eb2-43de-b885-71bc7b74e26e\") " pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.070547 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.070398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce309e35-7eb2-43de-b885-71bc7b74e26e-lib-modules\") pod \"perf-node-gather-daemonset-7qvrf\" (UID: \"ce309e35-7eb2-43de-b885-71bc7b74e26e\") " pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.070547 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.070462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce309e35-7eb2-43de-b885-71bc7b74e26e-podres\") pod \"perf-node-gather-daemonset-7qvrf\" (UID: \"ce309e35-7eb2-43de-b885-71bc7b74e26e\") " pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.070547 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.070497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce309e35-7eb2-43de-b885-71bc7b74e26e-proc\") pod \"perf-node-gather-daemonset-7qvrf\" (UID: \"ce309e35-7eb2-43de-b885-71bc7b74e26e\") " pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.171607 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.171570    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce309e35-7eb2-43de-b885-71bc7b74e26e-podres\") pod \"perf-node-gather-daemonset-7qvrf\" (UID: \"ce309e35-7eb2-43de-b885-71bc7b74e26e\") " pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.171775 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.171613    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce309e35-7eb2-43de-b885-71bc7b74e26e-proc\") pod \"perf-node-gather-daemonset-7qvrf\" (UID: \"ce309e35-7eb2-43de-b885-71bc7b74e26e\") " pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.171775 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.171667    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce309e35-7eb2-43de-b885-71bc7b74e26e-sys\") pod \"perf-node-gather-daemonset-7qvrf\" (UID: \"ce309e35-7eb2-43de-b885-71bc7b74e26e\") " pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.171775 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.171696    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-46mld\" (UniqueName: \"kubernetes.io/projected/ce309e35-7eb2-43de-b885-71bc7b74e26e-kube-api-access-46mld\") pod \"perf-node-gather-daemonset-7qvrf\" (UID: \"ce309e35-7eb2-43de-b885-71bc7b74e26e\") " pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.171775 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.171732    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce309e35-7eb2-43de-b885-71bc7b74e26e-podres\") pod \"perf-node-gather-daemonset-7qvrf\" (UID: \"ce309e35-7eb2-43de-b885-71bc7b74e26e\") " pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.171775 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.171749    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce309e35-7eb2-43de-b885-71bc7b74e26e-lib-modules\") pod \"perf-node-gather-daemonset-7qvrf\" (UID: \"ce309e35-7eb2-43de-b885-71bc7b74e26e\") " pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.171775 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.171751    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce309e35-7eb2-43de-b885-71bc7b74e26e-proc\") pod \"perf-node-gather-daemonset-7qvrf\" (UID: \"ce309e35-7eb2-43de-b885-71bc7b74e26e\") " pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.171775 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.171769    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce309e35-7eb2-43de-b885-71bc7b74e26e-sys\") pod \"perf-node-gather-daemonset-7qvrf\" (UID: \"ce309e35-7eb2-43de-b885-71bc7b74e26e\") " pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.172107 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.171909    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce309e35-7eb2-43de-b885-71bc7b74e26e-lib-modules\") pod \"perf-node-gather-daemonset-7qvrf\" (UID: \"ce309e35-7eb2-43de-b885-71bc7b74e26e\") " pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.179467 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.179441    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46mld\" (UniqueName: \"kubernetes.io/projected/ce309e35-7eb2-43de-b885-71bc7b74e26e-kube-api-access-46mld\") pod \"perf-node-gather-daemonset-7qvrf\" (UID: \"ce309e35-7eb2-43de-b885-71bc7b74e26e\") " pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.191857 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.191836    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n77fp_0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b/dns/0.log"
May 28 23:34:48.209231 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.209173    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n77fp_0f654df9-6c79-4a7f-9d6d-6933e9bdfa5b/kube-rbac-proxy/0.log"
May 28 23:34:48.264096 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.264066    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:48.266154 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.266131    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rhwv2_8856d7aa-86ca-4ef4-9d92-ceac3dd18292/dns-node-resolver/0.log"
May 28 23:34:48.391796 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.391757    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"]
May 28 23:34:48.394089 ip-10-0-143-104 kubenswrapper[2577]: W0528 23:34:48.394064    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podce309e35_7eb2_43de_b885_71bc7b74e26e.slice/crio-1c0956e33ce2c538171767c29d4cb84b1257d35792f13840590a977b38471d12 WatchSource:0}: Error finding container 1c0956e33ce2c538171767c29d4cb84b1257d35792f13840590a977b38471d12: Status 404 returned error can't find the container with id 1c0956e33ce2c538171767c29d4cb84b1257d35792f13840590a977b38471d12
May 28 23:34:48.395799 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.395780    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 23:34:48.421315 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.421285    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf" event={"ID":"ce309e35-7eb2-43de-b885-71bc7b74e26e","Type":"ContainerStarted","Data":"1c0956e33ce2c538171767c29d4cb84b1257d35792f13840590a977b38471d12"}
May 28 23:34:48.724423 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:48.724398    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cm8m8_8353cc62-2b97-4f45-8a3d-b871840ee8e8/node-ca/0.log"
May 28 23:34:49.426171 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:49.426131    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf" event={"ID":"ce309e35-7eb2-43de-b885-71bc7b74e26e","Type":"ContainerStarted","Data":"03ea84ce1d059081003433041711b0230b45949893203dd4489ca293ad65461e"}
May 28 23:34:49.426363 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:49.426270    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:49.442826 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:49.442780    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf" podStartSLOduration=2.4427670360000002 podStartE2EDuration="2.442767036s" podCreationTimestamp="2026-05-28 23:34:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 23:34:49.440527222 +0000 UTC m=+1962.633958486" watchObservedRunningTime="2026-05-28 23:34:49.442767036 +0000 UTC m=+1962.636198289"
May 28 23:34:49.707614 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:49.707532    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-47c8w_ba8f16cc-755b-4dc9-a1b5-3c33e7149c77/serve-healthcheck-canary/0.log"
May 28 23:34:50.123447 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:50.123389    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lx6p4_2841d669-9492-43c9-a8e4-a19d6c7d6b00/kube-rbac-proxy/0.log"
May 28 23:34:50.141377 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:50.141349    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lx6p4_2841d669-9492-43c9-a8e4-a19d6c7d6b00/exporter/0.log"
May 28 23:34:50.159469 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:50.159438    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lx6p4_2841d669-9492-43c9-a8e4-a19d6c7d6b00/extractor/0.log"
May 28 23:34:53.953794 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:53.953763    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-p2l28_175d6678-c45e-4aed-a2cb-d460d6d578cf/migrator/0.log"
May 28 23:34:53.972056 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:53.972027    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-p2l28_175d6678-c45e-4aed-a2cb-d460d6d578cf/graceful-termination/0.log"
May 28 23:34:55.196773 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:55.196688    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvj99_321b5d1e-083e-45f0-99c7-066f9ec958d2/kube-multus-additional-cni-plugins/0.log"
May 28 23:34:55.215346 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:55.215312    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvj99_321b5d1e-083e-45f0-99c7-066f9ec958d2/egress-router-binary-copy/0.log"
May 28 23:34:55.234040 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:55.234016    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvj99_321b5d1e-083e-45f0-99c7-066f9ec958d2/cni-plugins/0.log"
May 28 23:34:55.251565 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:55.251540    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvj99_321b5d1e-083e-45f0-99c7-066f9ec958d2/bond-cni-plugin/0.log"
May 28 23:34:55.269306 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:55.269281    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvj99_321b5d1e-083e-45f0-99c7-066f9ec958d2/routeoverride-cni/0.log"
May 28 23:34:55.286732 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:55.286703    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvj99_321b5d1e-083e-45f0-99c7-066f9ec958d2/whereabouts-cni-bincopy/0.log"
May 28 23:34:55.304143 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:55.304111    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zvj99_321b5d1e-083e-45f0-99c7-066f9ec958d2/whereabouts-cni/0.log"
May 28 23:34:55.334018 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:55.333992    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dfz7l_98206e77-e8fd-42da-bfb0-3fd1779ddd2e/kube-multus/0.log"
May 28 23:34:55.373718 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:55.373690    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mwvzm_0a871ef7-90af-4a70-96ae-7df74d7ca072/network-metrics-daemon/0.log"
May 28 23:34:55.390748 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:55.390730    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mwvzm_0a871ef7-90af-4a70-96ae-7df74d7ca072/kube-rbac-proxy/0.log"
May 28 23:34:55.439778 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:55.439755    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vppqb/perf-node-gather-daemonset-7qvrf"
May 28 23:34:56.736517 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:56.736491    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w59bt_364d7926-6000-4731-9989-15e6d74a44c6/ovn-controller/0.log"
May 28 23:34:56.763619 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:56.763596    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w59bt_364d7926-6000-4731-9989-15e6d74a44c6/ovn-acl-logging/0.log"
May 28 23:34:56.782885 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:56.782847    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w59bt_364d7926-6000-4731-9989-15e6d74a44c6/kube-rbac-proxy-node/0.log"
May 28 23:34:56.804317 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:56.804286    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w59bt_364d7926-6000-4731-9989-15e6d74a44c6/kube-rbac-proxy-ovn-metrics/0.log"
May 28 23:34:56.823164 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:56.823146    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w59bt_364d7926-6000-4731-9989-15e6d74a44c6/northd/0.log"
May 28 23:34:56.843264 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:56.843201    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w59bt_364d7926-6000-4731-9989-15e6d74a44c6/nbdb/0.log"
May 28 23:34:56.863203 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:56.863182    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w59bt_364d7926-6000-4731-9989-15e6d74a44c6/sbdb/0.log"
May 28 23:34:56.966344 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:56.966320    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w59bt_364d7926-6000-4731-9989-15e6d74a44c6/ovnkube-controller/0.log"
May 28 23:34:57.885758 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:57.885680    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7f44b6dd87-vfhc7_e6564129-6c7b-4789-aca5-0ce1eb814173/check-endpoints/0.log"
May 28 23:34:57.905368 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:57.905339    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2kktk_9c10eeed-3826-4345-9e9b-6b8d25111763/network-check-target-container/0.log"
May 28 23:34:58.838003 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:58.837977    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-w4wxq_42519605-09ec-4431-9c90-a67e86597a9e/iptables-alerter/0.log"
May 28 23:34:59.364947 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:34:59.364856    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-drr5k_0c1000e8-ffa7-412e-a0ba-d0717ba2bbbb/tuned/0.log"
May 28 23:35:00.849457 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:35:00.849427    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-x4gsd_c93652d2-0a39-40de-ae3c-e31fa167e356/cluster-samples-operator/0.log"
May 28 23:35:00.865566 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:35:00.865525    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-x4gsd_c93652d2-0a39-40de-ae3c-e31fa167e356/cluster-samples-operator-watch/0.log"
May 28 23:35:01.973830 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:35:01.973799    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-67d667c7d-fsdnz_9c347fe6-bfc7-4d0a-8974-92bb5edc26f9/service-ca-controller/0.log"
May 28 23:35:02.281432 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:35:02.281398    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-26l5v_9559d46d-6176-4e25-a648-d11e8f9e99a6/csi-driver/0.log"
May 28 23:35:02.299546 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:35:02.299514    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-26l5v_9559d46d-6176-4e25-a648-d11e8f9e99a6/csi-node-driver-registrar/0.log"
May 28 23:35:02.317443 ip-10-0-143-104 kubenswrapper[2577]: I0528 23:35:02.317418    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-26l5v_9559d46d-6176-4e25-a648-d11e8f9e99a6/csi-liveness-probe/0.log"