May 28 07:59:07.415799 ip-10-0-136-248 systemd[1]: Starting Kubernetes Kubelet...
May 28 07:59:07.939657 ip-10-0-136-248 kubenswrapper[2551]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 07:59:07.939657 ip-10-0-136-248 kubenswrapper[2551]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 28 07:59:07.939657 ip-10-0-136-248 kubenswrapper[2551]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 07:59:07.939657 ip-10-0-136-248 kubenswrapper[2551]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 28 07:59:07.939657 ip-10-0-136-248 kubenswrapper[2551]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 28 07:59:07.941488 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.941405    2551 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 28 07:59:07.948964 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.948937    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 07:59:07.948964 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.948957    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 07:59:07.948964 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.948961    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 07:59:07.948964 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.948965    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 07:59:07.948964 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.948967    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 07:59:07.948964 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.948971    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 07:59:07.948964 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.948974    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.948979    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.948983    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.948986    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.948989    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.948992    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.948994    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.948998    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949000    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949003    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949006    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949009    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949013    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949016    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949019    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949022    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949024    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949027    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949031    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 07:59:07.949222 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949033    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949036    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949042    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949045    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949047    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949050    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949052    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949054    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949057    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949060    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949062    2551 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949065    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949067    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949070    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949072    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949076    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949078    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949081    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949084    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949086    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 07:59:07.949698 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949089    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949092    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949095    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949097    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949100    2551 feature_gate.go:328] unrecognized feature gate: Example
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949103    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949105    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949107    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949110    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949113    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949116    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949118    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949121    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949124    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949126    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949129    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949132    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949134    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949137    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949139    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 07:59:07.950196 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949142    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949144    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949147    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949151    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949153    2551 feature_gate.go:328] unrecognized feature gate: Example2
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949155    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949158    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949162    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949164    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949167    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949170    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949173    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949176    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949179    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949181    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949184    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949186    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949188    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949191    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949193    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 07:59:07.950744 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.949196    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951084    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951092    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951095    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951098    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951100    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951103    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951106    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951108    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951111    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951114    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951116    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951119    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951121    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951124    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951126    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951130    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951132    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951134    2551 feature_gate.go:328] unrecognized feature gate: Example
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951137    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 07:59:07.951221 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951139    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951142    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951144    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951147    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951149    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951152    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951154    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951157    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951159    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951162    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951164    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951168    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951171    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951173    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951177    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951180    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951183    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951185    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951188    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951191    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 07:59:07.951714 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951193    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951196    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951199    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951201    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951203    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951206    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951208    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951212    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951216    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951219    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951221    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951224    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951227    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951229    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951232    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951235    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951237    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951240    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951243    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951245    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 07:59:07.952203 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951248    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951250    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951253    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951255    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951257    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951260    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951263    2551 feature_gate.go:328] unrecognized feature gate: Example2
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951266    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951269    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951271    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951274    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951277    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951280    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951282    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951285    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951287    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951290    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951294    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951297    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951299    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 07:59:07.952706 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951302    2551 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951304    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951308    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951310    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951313    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951315    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.951318    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951414    2551 flags.go:64] FLAG: --address="0.0.0.0"
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951424    2551 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951431    2551 flags.go:64] FLAG: --anonymous-auth="true"
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951436    2551 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951441    2551 flags.go:64] FLAG: --authentication-token-webhook="false"
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951444    2551 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951449    2551 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951453    2551 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951456    2551 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951459    2551 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951463    2551 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951466    2551 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951469    2551 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951473    2551 flags.go:64] FLAG: --cgroup-root=""
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951476    2551 flags.go:64] FLAG: --cgroups-per-qos="true"
May 28 07:59:07.953192 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951478    2551 flags.go:64] FLAG: --client-ca-file=""
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951481    2551 flags.go:64] FLAG: --cloud-config=""
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951484    2551 flags.go:64] FLAG: --cloud-provider="external"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951487    2551 flags.go:64] FLAG: --cluster-dns="[]"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951491    2551 flags.go:64] FLAG: --cluster-domain=""
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951494    2551 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951497    2551 flags.go:64] FLAG: --config-dir=""
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951499    2551 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951503    2551 flags.go:64] FLAG: --container-log-max-files="5"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951507    2551 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951510    2551 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951513    2551 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951516    2551 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951519    2551 flags.go:64] FLAG: --contention-profiling="false"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951522    2551 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951525    2551 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951528    2551 flags.go:64] FLAG: --cpu-manager-policy="none"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951531    2551 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951535    2551 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951538    2551 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951541    2551 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951544    2551 flags.go:64] FLAG: --enable-load-reader="false"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951548    2551 flags.go:64] FLAG: --enable-server="true"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951551    2551 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951556    2551 flags.go:64] FLAG: --event-burst="100"
May 28 07:59:07.953734 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951559    2551 flags.go:64] FLAG: --event-qps="50"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951562    2551 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951566    2551 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951569    2551 flags.go:64] FLAG: --eviction-hard=""
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951573    2551 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951576    2551 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951578    2551 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951581    2551 flags.go:64] FLAG: --eviction-soft=""
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951584    2551 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951587    2551 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951590    2551 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951593    2551 flags.go:64] FLAG: --experimental-mounter-path=""
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951596    2551 flags.go:64] FLAG: --fail-cgroupv1="false"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951599    2551 flags.go:64] FLAG: --fail-swap-on="true"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951602    2551 flags.go:64] FLAG: --feature-gates=""
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951606    2551 flags.go:64] FLAG: --file-check-frequency="20s"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951608    2551 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951611    2551 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951614    2551 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951617    2551 flags.go:64] FLAG: --healthz-port="10248"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951621    2551 flags.go:64] FLAG: --help="false"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951623    2551 flags.go:64] FLAG: --hostname-override="ip-10-0-136-248.ec2.internal"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951627    2551 flags.go:64] FLAG: --housekeeping-interval="10s"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951630    2551 flags.go:64] FLAG: --http-check-frequency="20s"
May 28 07:59:07.954358 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951633    2551 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951636    2551 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951639    2551 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951643    2551 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951646    2551 flags.go:64] FLAG: --image-service-endpoint=""
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951649    2551 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951653    2551 flags.go:64] FLAG: --kube-api-burst="100"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951656    2551 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951659    2551 flags.go:64] FLAG: --kube-api-qps="50"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951663    2551 flags.go:64] FLAG: --kube-reserved=""
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951666    2551 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951669    2551 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951672    2551 flags.go:64] FLAG: --kubelet-cgroups=""
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951674    2551 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951677    2551 flags.go:64] FLAG: --lock-file=""
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951680    2551 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951683    2551 flags.go:64] FLAG: --log-flush-frequency="5s"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951686    2551 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951691    2551 flags.go:64] FLAG: --log-json-split-stream="false"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951694    2551 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951697    2551 flags.go:64] FLAG: --log-text-split-stream="false"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951700    2551 flags.go:64] FLAG: --logging-format="text"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951703    2551 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951706    2551 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 28 07:59:07.955317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951709    2551 flags.go:64] FLAG: --manifest-url=""
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951712    2551 flags.go:64] FLAG: --manifest-url-header=""
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951716    2551 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951720    2551 flags.go:64] FLAG: --max-open-files="1000000"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951728    2551 flags.go:64] FLAG: --max-pods="110"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951731    2551 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951734    2551 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951737    2551 flags.go:64] FLAG: --memory-manager-policy="None"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951740    2551 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951743    2551 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951746    2551 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951748    2551 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951756    2551 flags.go:64] FLAG: --node-status-max-images="50"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951759    2551 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951762    2551 flags.go:64] FLAG: --oom-score-adj="-999"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951766    2551 flags.go:64] FLAG: --pod-cidr=""
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951769    2551 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951775    2551 flags.go:64] FLAG: --pod-manifest-path=""
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951778    2551 flags.go:64] FLAG: --pod-max-pids="-1"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951782    2551 flags.go:64] FLAG: --pods-per-core="0"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951785    2551 flags.go:64] FLAG: --port="10250"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951788    2551 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951791    2551 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d3f8f0d6060b8a42"
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951795    2551 flags.go:64] FLAG: --qos-reserved=""
May 28 07:59:07.955959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951798    2551 flags.go:64] FLAG: --read-only-port="10255"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951800    2551 flags.go:64] FLAG: --register-node="true"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951803    2551 flags.go:64] FLAG: --register-schedulable="true"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951806    2551 flags.go:64] FLAG: --register-with-taints=""
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951810    2551 flags.go:64] FLAG: --registry-burst="10"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951812    2551 flags.go:64] FLAG: --registry-qps="5"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951815    2551 flags.go:64] FLAG: --reserved-cpus=""
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951818    2551 flags.go:64] FLAG: --reserved-memory=""
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951822    2551 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951825    2551 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951828    2551 flags.go:64] FLAG: --rotate-certificates="false"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951831    2551 flags.go:64] FLAG: --rotate-server-certificates="false"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951834    2551 flags.go:64] FLAG: --runonce="false"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951837    2551 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951841    2551 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951843    2551 flags.go:64] FLAG: --seccomp-default="false"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951846    2551 flags.go:64] FLAG: --serialize-image-pulls="true"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951849    2551 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951852    2551 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951855    2551 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951858    2551 flags.go:64] FLAG: --storage-driver-password="root"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951861    2551 flags.go:64] FLAG: --storage-driver-secure="false"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951864    2551 flags.go:64] FLAG: --storage-driver-table="stats"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951866    2551 flags.go:64] FLAG: --storage-driver-user="root"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951870    2551 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951873    2551 flags.go:64] FLAG: --sync-frequency="1m0s"
May 28 07:59:07.956562 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951876    2551 flags.go:64] FLAG: --system-cgroups=""
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951878    2551 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951884    2551 flags.go:64] FLAG: --system-reserved-cgroup=""
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951886    2551 flags.go:64] FLAG: --tls-cert-file=""
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951889    2551 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951893    2551 flags.go:64] FLAG: --tls-min-version=""
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951895    2551 flags.go:64] FLAG: --tls-private-key-file=""
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951898    2551 flags.go:64] FLAG: --topology-manager-policy="none"
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951901    2551 flags.go:64] FLAG: --topology-manager-policy-options=""
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951904    2551 flags.go:64] FLAG: --topology-manager-scope="container"
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951907    2551 flags.go:64] FLAG: --v="2"
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951911    2551 flags.go:64] FLAG: --version="false"
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951914    2551 flags.go:64] FLAG: --vmodule=""
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951919    2551 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.951922    2551 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952009    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952012    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952015    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952019    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952022    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952025    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952027    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952030    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 07:59:07.957197 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952033    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952036    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952038    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952041    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952044    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952046    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952049    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952054    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952058    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952061    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952064    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952067    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952070    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952072    2551 feature_gate.go:328] unrecognized feature gate: Example2
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952075    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952078    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952080    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952083    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952085    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 07:59:07.957752 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952088    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952090    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952092    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952095    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952098    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952100    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952103    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952106    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952110    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952113    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952116    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952118    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952121    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952124    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952127    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952129    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952132    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952135    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952138    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952141    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 07:59:07.958244 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952143    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952146    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952149    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952151    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952154    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952156    2551 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952159    2551 feature_gate.go:328] unrecognized feature gate: Example
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952161    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952164    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952166    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952169    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952171    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952173    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952176    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952178    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952181    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952183    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952186    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952188    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952190    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 07:59:07.958751 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952193    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952196    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952198    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952201    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952204    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952207    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952209    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952212    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952215    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952217    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952220    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952223    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952225    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952228    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952230    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952233    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952235    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952238    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 07:59:07.959300 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.952240    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 07:59:07.959778 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.953256    2551 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 07:59:07.961206 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.961187    2551 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 28 07:59:07.961247 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.961208    2551 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 28 07:59:07.961275 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961257    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 07:59:07.961275 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961262    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 07:59:07.961275 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961266    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 07:59:07.961275 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961269    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 07:59:07.961275 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961272    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 07:59:07.961275 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961275    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 07:59:07.961275 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961279    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961281    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961284    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961287    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961289    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961292    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961294    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961297    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961300    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961302    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961305    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961308    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961310    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961313    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961316    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961318    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961321    2551 feature_gate.go:328] unrecognized feature gate: Example2
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961324    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961326    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961329    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 07:59:07.961466 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961331    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961333    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961336    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961338    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961341    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961345    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961347    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961350    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961352    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961355    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961358    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961360    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961363    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961365    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961369    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961372    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961374    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961377    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961391    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961395    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 07:59:07.961949 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961398    2551 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961400    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961403    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961405    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961408    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961411    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961414    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961416    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961419    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961421    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961424    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961426    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961429    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961431    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961434    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961436    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961439    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961442    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961445    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961447    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 07:59:07.962463 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961450    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961453    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961455    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961458    2551 feature_gate.go:328] unrecognized feature gate: Example
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961460    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961464    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961469    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961473    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961475    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961479    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961481    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961484    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961487    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961491    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961493    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961496    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961499    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961503    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961507    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 07:59:07.962946 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961510    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.961515    2551 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961622    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961627    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961630    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961633    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961636    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961639    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961642    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961645    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961648    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961650    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961661    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961664    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961667    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961670    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 28 07:59:07.963505 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961672    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961675    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961678    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961680    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961683    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961686    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961689    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961691    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961694    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961697    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961699    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961702    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961704    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961707    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961709    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961712    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961714    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961717    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961719    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961722    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 28 07:59:07.963914 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961724    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961727    2551 feature_gate.go:328] unrecognized feature gate: Example
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961729    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961732    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961734    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961736    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961739    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961741    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961744    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961748    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961753    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961756    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961759    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961762    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961764    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961768    2551 feature_gate.go:328] unrecognized feature gate: Example2
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961770    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961773    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961776    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 28 07:59:07.964444 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961778    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961781    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961783    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961786    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961788    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961791    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961793    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961795    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961798    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961801    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961804    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961808    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961811    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961814    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961817    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961819    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961822    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961824    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961827    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
May 28 07:59:07.964897 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961829    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 28 07:59:07.965356 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961832    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 28 07:59:07.965356 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961835    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 28 07:59:07.965356 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961838    2551 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 28 07:59:07.965356 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961841    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 28 07:59:07.965356 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961843    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 28 07:59:07.965356 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961846    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 28 07:59:07.965356 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961848    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 28 07:59:07.965356 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961851    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 28 07:59:07.965356 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961853    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 28 07:59:07.965356 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961856    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 28 07:59:07.965356 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961858    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 28 07:59:07.965356 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961860    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 28 07:59:07.965356 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:07.961863    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 28 07:59:07.965356 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.961868    2551 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 28 07:59:07.965356 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.962911    2551 server.go:962] "Client rotation is on, will bootstrap in background"
May 28 07:59:07.965755 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.965260    2551 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 28 07:59:07.966429 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.966417    2551 server.go:1019] "Starting client certificate rotation"
May 28 07:59:07.966534 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.966517    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 28 07:59:07.966572 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.966554    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 28 07:59:07.993057 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.993036    2551 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 28 07:59:07.995780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:07.995759    2551 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 28 07:59:08.008611 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.008591    2551 log.go:25] "Validated CRI v1 runtime API"
May 28 07:59:08.016791 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.016770    2551 log.go:25] "Validated CRI v1 image API"
May 28 07:59:08.018364 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.018345    2551 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 28 07:59:08.025000 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.024982    2551 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 28 07:59:08.026423 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.026405    2551 fs.go:135] Filesystem UUIDs: map[0d01874a-04f7-4ef9-be32-f6a2d085e372:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8d09e245-3cd0-4ac4-9ef0-140026039678:/dev/nvme0n1p3]
May 28 07:59:08.026471 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.026423    2551 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 28 07:59:08.030735 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.030622    2551 manager.go:217] Machine: {Timestamp:2026-05-28 07:59:08.030109024 +0000 UTC m=+0.470181304 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100636 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec252a231bf977361b6e40f587b60a4b SystemUUID:ec252a23-1bf9-7736-1b6e-40f587b60a4b BootID:0b00bc54-34a8-4671-bda0-e0acf4e106b1 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5b:94:0c:ae:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5b:94:0c:ae:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:f6:d5:40:49:2b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 28 07:59:08.031606 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.031596    2551 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 28 07:59:08.031692 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.031680    2551 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 28 07:59:08.033879 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.033846    2551 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 28 07:59:08.034056 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.033882    2551 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-248.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 28 07:59:08.034144 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.034071    2551 topology_manager.go:138] "Creating topology manager with none policy"
May 28 07:59:08.034144 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.034083    2551 container_manager_linux.go:306] "Creating device plugin manager"
May 28 07:59:08.034144 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.034093    2551 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 28 07:59:08.035027 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.035013    2551 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 28 07:59:08.037186 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.037173    2551 state_mem.go:36] "Initialized new in-memory state store"
May 28 07:59:08.037324 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.037313    2551 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 28 07:59:08.040289 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.040278    2551 kubelet.go:491] "Attempting to sync node with API server"
May 28 07:59:08.040353 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.040295    2551 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 28 07:59:08.040353 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.040312    2551 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 28 07:59:08.040353 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.040325    2551 kubelet.go:397] "Adding apiserver pod source"
May 28 07:59:08.040353 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.040337    2551 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 28 07:59:08.041573 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.041560    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 28 07:59:08.041643 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.041583    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 28 07:59:08.042682 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.042664    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vptdx"
May 28 07:59:08.045150 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.045132    2551 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 28 07:59:08.047329 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.047316    2551 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 28 07:59:08.049539 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.049522    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vptdx"
May 28 07:59:08.049597 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.049587    2551 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 28 07:59:08.049636 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.049602    2551 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 28 07:59:08.049636 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.049608    2551 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 28 07:59:08.049636 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.049613    2551 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 28 07:59:08.049636 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.049623    2551 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 28 07:59:08.049636 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.049629    2551 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 28 07:59:08.049636 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.049635    2551 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 28 07:59:08.049802 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.049641    2551 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 28 07:59:08.049802 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.049648    2551 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 28 07:59:08.049802 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.049655    2551 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 28 07:59:08.049802 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.049671    2551 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 28 07:59:08.049802 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.049676    2551 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 28 07:59:08.051797 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.051785    2551 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 28 07:59:08.051834 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.051797    2551 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 28 07:59:08.054831 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.054815    2551 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 07:59:08.055002 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.054990    2551 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 28 07:59:08.055101 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.055026    2551 server.go:1295] "Started kubelet"
May 28 07:59:08.055877 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.055700    2551 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 28 07:59:08.056004 ip-10-0-136-248 systemd[1]: Started Kubernetes Kubelet.
May 28 07:59:08.056177 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.056158    2551 server_v1.go:47] "podresources" method="list" useActivePods=true
May 28 07:59:08.056297 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.055893    2551 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 28 07:59:08.060963 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.057701    2551 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 28 07:59:08.061940 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.061919    2551 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 07:59:08.063152 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.063135    2551 server.go:317] "Adding debug handlers to kubelet server"
May 28 07:59:08.063290 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.063270    2551 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-136-248.ec2.internal" not found
May 28 07:59:08.065923 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:08.065903    2551 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 28 07:59:08.067956 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.067943    2551 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 28 07:59:08.068059 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.067968    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 28 07:59:08.068276 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.068262    2551 volume_manager.go:295] "The desired_state_of_world populator starts"
May 28 07:59:08.068354 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.068278    2551 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 28 07:59:08.068354 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.068304    2551 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 28 07:59:08.068460 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.068398    2551 reconstruct.go:97] "Volume reconstruction finished"
May 28 07:59:08.068460 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.068415    2551 reconciler.go:26] "Reconciler: start to sync state"
May 28 07:59:08.069868 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:08.069159    2551 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-248.ec2.internal\" not found"
May 28 07:59:08.070078 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.070062    2551 factory.go:55] Registering systemd factory
May 28 07:59:08.070165 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.070081    2551 factory.go:223] Registration of the systemd container factory successfully
May 28 07:59:08.070357 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.070342    2551 factory.go:153] Registering CRI-O factory
May 28 07:59:08.070357 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.070360    2551 factory.go:223] Registration of the crio container factory successfully
May 28 07:59:08.070515 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.070432    2551 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 28 07:59:08.070515 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.070449    2551 factory.go:103] Registering Raw factory
May 28 07:59:08.070515 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.070460    2551 manager.go:1196] Started watching for new ooms in manager
May 28 07:59:08.070904 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.070884    2551 manager.go:319] Starting recovery of all containers
May 28 07:59:08.070996 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.070969    2551 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 07:59:08.072517 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:08.072324    2551 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-248.ec2.internal\" not found" node="ip-10-0-136-248.ec2.internal"
May 28 07:59:08.078211 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.078194    2551 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-136-248.ec2.internal" not found
May 28 07:59:08.081447 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.081428    2551 manager.go:324] Recovery completed
May 28 07:59:08.085518 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.085506    2551 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 07:59:08.088369 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.088352    2551 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-248.ec2.internal" event="NodeHasSufficientMemory"
May 28 07:59:08.088445 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.088394    2551 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-248.ec2.internal" event="NodeHasNoDiskPressure"
May 28 07:59:08.088445 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.088406    2551 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-248.ec2.internal" event="NodeHasSufficientPID"
May 28 07:59:08.088886 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.088868    2551 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 28 07:59:08.088886 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.088883    2551 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 28 07:59:08.088988 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.088899    2551 state_mem.go:36] "Initialized new in-memory state store"
May 28 07:59:08.092811 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.092797    2551 policy_none.go:49] "None policy: Start"
May 28 07:59:08.092874 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.092814    2551 memory_manager.go:186] "Starting memorymanager" policy="None"
May 28 07:59:08.092874 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.092825    2551 state_mem.go:35] "Initializing new in-memory state store"
May 28 07:59:08.131052 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.131031    2551 manager.go:341] "Starting Device Plugin manager"
May 28 07:59:08.148441 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:08.131073    2551 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 28 07:59:08.148441 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.131084    2551 server.go:85] "Starting device plugin registration server"
May 28 07:59:08.148441 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.131317    2551 eviction_manager.go:189] "Eviction manager: starting control loop"
May 28 07:59:08.148441 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.131327    2551 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 28 07:59:08.148441 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.131473    2551 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 28 07:59:08.148441 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.131554    2551 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 28 07:59:08.148441 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.131562    2551 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 28 07:59:08.148441 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:08.132061    2551 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 28 07:59:08.148441 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:08.132106    2551 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-248.ec2.internal\" not found"
May 28 07:59:08.148441 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.133439    2551 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-136-248.ec2.internal" not found
May 28 07:59:08.188856 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.188821    2551 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 28 07:59:08.190088 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.190036    2551 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 28 07:59:08.190088 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.190076    2551 status_manager.go:230] "Starting to sync pod status with apiserver"
May 28 07:59:08.190212 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.190097    2551 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 28 07:59:08.190212 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.190107    2551 kubelet.go:2451] "Starting kubelet main sync loop"
May 28 07:59:08.190212 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:08.190152    2551 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 28 07:59:08.192320 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.192303    2551 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 07:59:08.231934 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.231906    2551 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 28 07:59:08.233376 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.233362    2551 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-248.ec2.internal" event="NodeHasSufficientMemory"
May 28 07:59:08.233447 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.233408    2551 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-248.ec2.internal" event="NodeHasNoDiskPressure"
May 28 07:59:08.233447 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.233419    2551 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-248.ec2.internal" event="NodeHasSufficientPID"
May 28 07:59:08.233447 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.233440    2551 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-248.ec2.internal"
May 28 07:59:08.244715 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.244687    2551 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-248.ec2.internal"
May 28 07:59:08.290805 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.290739    2551 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-248.ec2.internal"]
May 28 07:59:08.292838 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.292820    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-248.ec2.internal"
May 28 07:59:08.292926 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.292827    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal"
May 28 07:59:08.319944 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.319924    2551 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-248.ec2.internal"
May 28 07:59:08.324230 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.324215    2551 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal"
May 28 07:59:08.332144 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.332129    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 28 07:59:08.334600 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.334580    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 28 07:59:08.469662 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.469633    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc0a0cad24c1a6672f2575bc0047f126-config\") pod \"kube-apiserver-proxy-ip-10-0-136-248.ec2.internal\" (UID: \"fc0a0cad24c1a6672f2575bc0047f126\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-248.ec2.internal"
May 28 07:59:08.469802 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.469662    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5062c11c04938e57cbece55012367eff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal\" (UID: \"5062c11c04938e57cbece55012367eff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal"
May 28 07:59:08.469802 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.469686    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5062c11c04938e57cbece55012367eff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal\" (UID: \"5062c11c04938e57cbece55012367eff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal"
May 28 07:59:08.570098 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.570064    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5062c11c04938e57cbece55012367eff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal\" (UID: \"5062c11c04938e57cbece55012367eff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal"
May 28 07:59:08.570098 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.570101    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5062c11c04938e57cbece55012367eff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal\" (UID: \"5062c11c04938e57cbece55012367eff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal"
May 28 07:59:08.570297 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.570125    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc0a0cad24c1a6672f2575bc0047f126-config\") pod \"kube-apiserver-proxy-ip-10-0-136-248.ec2.internal\" (UID: \"fc0a0cad24c1a6672f2575bc0047f126\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-248.ec2.internal"
May 28 07:59:08.570297 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.570176    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5062c11c04938e57cbece55012367eff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal\" (UID: \"5062c11c04938e57cbece55012367eff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal"
May 28 07:59:08.570297 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.570202    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc0a0cad24c1a6672f2575bc0047f126-config\") pod \"kube-apiserver-proxy-ip-10-0-136-248.ec2.internal\" (UID: \"fc0a0cad24c1a6672f2575bc0047f126\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-248.ec2.internal"
May 28 07:59:08.570297 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.570177    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5062c11c04938e57cbece55012367eff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal\" (UID: \"5062c11c04938e57cbece55012367eff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal"
May 28 07:59:08.636224 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.636192    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal"
May 28 07:59:08.636293 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.636249    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-248.ec2.internal"
May 28 07:59:08.966188 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.966157    2551 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 28 07:59:08.966793 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.966342    2551 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 28 07:59:08.966793 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.966341    2551 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 28 07:59:08.966793 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:08.966342    2551 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 28 07:59:09.041447 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.041419    2551 apiserver.go:52] "Watching apiserver"
May 28 07:59:09.049557 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.049526    2551 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 28 07:59:09.049865 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.049833    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-wcbtq","openshift-network-operator/iptables-alerter-p6zng","openshift-image-registry/node-ca-q9tbf","openshift-multus/multus-fjgcp","openshift-multus/network-metrics-daemon-dfswh","openshift-network-diagnostics/network-check-target-t782k","openshift-ovn-kubernetes/ovnkube-node-vvpbp","kube-system/konnectivity-agent-nfsn2","kube-system/kube-apiserver-proxy-ip-10-0-136-248.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj","openshift-cluster-node-tuning-operator/tuned-lfc7s","openshift-dns/node-resolver-8vcnc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal"]
May 28 07:59:09.051496 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.051471    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-27 07:54:08 +0000 UTC" deadline="2028-02-12 21:32:02.013558898 +0000 UTC"
May 28 07:59:09.051558 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.051496    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15013h32m52.962065268s"
May 28 07:59:09.054485 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.054464    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.054586 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.054472    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p6zng"
May 28 07:59:09.056828 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.056808    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q9tbf"
May 28 07:59:09.056922 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.056907    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qts89\""
May 28 07:59:09.057098 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.057075    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 28 07:59:09.057098 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.057077    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 28 07:59:09.057218 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.057121    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 28 07:59:09.057269 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.057254    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 28 07:59:09.057316 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.057255    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 28 07:59:09.057316 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.057309    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9977k\""
May 28 07:59:09.057995 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.057981    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 28 07:59:09.058049 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.057999    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 28 07:59:09.058049 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.057999    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 28 07:59:09.058843 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.058824    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.058963 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.058946    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 28 07:59:09.059024 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.059001    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 28 07:59:09.059248 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.059232    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 28 07:59:09.059828 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.059808    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2jjw6\""
May 28 07:59:09.060946 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.060922    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 28 07:59:09.061035 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.060962    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:09.061264 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.061248    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-85gtk\""
May 28 07:59:09.061409 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:09.061244    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:09.064539 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.064523    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:09.064613 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:09.064582    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:09.066707 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.066688    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.068757 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.068729    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nfsn2"
May 28 07:59:09.068757 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.068740    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 28 07:59:09.069074 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.069053    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 28 07:59:09.069169 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.069144    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hpls4\""
May 28 07:59:09.070087 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.070070    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 28 07:59:09.070194 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.070070    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 28 07:59:09.070194 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.070073    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 28 07:59:09.070194 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.070073    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 28 07:59:09.070971 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.070951    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 28 07:59:09.071569 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.071551    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.072360 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.072344    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m5nj7\""
May 28 07:59:09.072607 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.072591    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 28 07:59:09.072695 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.072677    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 28 07:59:09.073438 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073421    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-multus-socket-dir-parent\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.073534 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073447    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-run-ovn\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.073534 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073469    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-cni-netd\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.073534 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073498    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a310a270-ab14-4bfc-ade4-123b9ee11db4-konnectivity-ca\") pod \"konnectivity-agent-nfsn2\" (UID: \"a310a270-ab14-4bfc-ade4-123b9ee11db4\") " pod="kube-system/konnectivity-agent-nfsn2"
May 28 07:59:09.073534 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073521    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-run-k8s-cni-cncf-io\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.073730 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073543    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/520071c5-2f56-457c-9fc5-d833415d3248-ovn-node-metrics-cert\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.073730 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073568    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-sys-fs\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.073730 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073630    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-system-cni-dir\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.073730 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073656    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-run-openvswitch\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.073730 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073659    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 28 07:59:09.073730 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073677    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 28 07:59:09.073730 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073681    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/520071c5-2f56-457c-9fc5-d833415d3248-ovnkube-config\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.073730 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073711    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 28 07:59:09.073730 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073722    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-registration-dir\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.074066 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073750    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-os-release\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.074066 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073752    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pclqd\""
May 28 07:59:09.074066 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073768    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-var-lib-cni-multus\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.074066 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073783    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a310a270-ab14-4bfc-ade4-123b9ee11db4-agent-certs\") pod \"konnectivity-agent-nfsn2\" (UID: \"a310a270-ab14-4bfc-ade4-123b9ee11db4\") " pod="kube-system/konnectivity-agent-nfsn2"
May 28 07:59:09.074066 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073802    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-log-socket\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.074066 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073827    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-run-ovn-kubernetes\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.074066 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073846    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9-iptables-alerter-script\") pod \"iptables-alerter-p6zng\" (UID: \"1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9\") " pod="openshift-network-operator/iptables-alerter-p6zng"
May 28 07:59:09.074066 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073869    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-var-lib-cni-bin\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.074066 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073891    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49ad95da-9ff8-462b-86aa-6f6eedf9ae3e-serviceca\") pod \"node-ca-q9tbf\" (UID: \"49ad95da-9ff8-462b-86aa-6f6eedf9ae3e\") " pod="openshift-image-registry/node-ca-q9tbf"
May 28 07:59:09.074066 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073937    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-etc-kubernetes\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.074066 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.073969    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-etc-openvswitch\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.074066 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074008    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-run-netns\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.074066 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074030    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-var-lib-kubelet\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.074066 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074067    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-kubelet\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.074483 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074097    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-run-netns\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.074483 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074113    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-node-log\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.074483 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074132    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9-host-slash\") pod \"iptables-alerter-p6zng\" (UID: \"1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9\") " pod="openshift-network-operator/iptables-alerter-p6zng"
May 28 07:59:09.074483 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074155    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl7dc\" (UniqueName: \"kubernetes.io/projected/49ad95da-9ff8-462b-86aa-6f6eedf9ae3e-kube-api-access-dl7dc\") pod \"node-ca-q9tbf\" (UID: \"49ad95da-9ff8-462b-86aa-6f6eedf9ae3e\") " pod="openshift-image-registry/node-ca-q9tbf"
May 28 07:59:09.074483 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074212    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs\") pod \"network-metrics-daemon-dfswh\" (UID: \"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd\") " pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:09.074483 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074245    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-run-systemd\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.074483 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074270    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpxg8\" (UniqueName: \"kubernetes.io/projected/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-kube-api-access-vpxg8\") pod \"network-metrics-daemon-dfswh\" (UID: \"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd\") " pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:09.074483 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074296    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdtts\" (UniqueName: \"kubernetes.io/projected/6fcecffb-0699-457c-a17f-be7801546dd8-kube-api-access-tdtts\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.074483 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074318    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-multus-cni-dir\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.074483 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074342    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-run-multus-certs\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.074483 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074375    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brpl\" (UniqueName: \"kubernetes.io/projected/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-kube-api-access-9brpl\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.074483 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074422    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-slash\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.074483 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074445    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64ffc18e-c32d-4d14-869f-fe837ec83037-os-release\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.074483 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074467    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-systemd-units\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074491    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-var-lib-openvswitch\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074497    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074514    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-hostroot\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074543    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/520071c5-2f56-457c-9fc5-d833415d3248-env-overrides\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074567    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-cni-bin\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074594    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vphsz\" (UniqueName: \"kubernetes.io/projected/520071c5-2f56-457c-9fc5-d833415d3248-kube-api-access-vphsz\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074646    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn7xf\" (UniqueName: \"kubernetes.io/projected/1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9-kube-api-access-rn7xf\") pod \"iptables-alerter-p6zng\" (UID: \"1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9\") " pod="openshift-network-operator/iptables-alerter-p6zng"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074682    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-socket-dir\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074707    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074732    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64ffc18e-c32d-4d14-869f-fe837ec83037-cni-binary-copy\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074756    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074782    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-device-dir\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074804    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64ffc18e-c32d-4d14-869f-fe837ec83037-cnibin\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074845    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64ffc18e-c32d-4d14-869f-fe837ec83037-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074899    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64ffc18e-c32d-4d14-869f-fe837ec83037-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.075067 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074927    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn2gn\" (UniqueName: \"kubernetes.io/projected/64ffc18e-c32d-4d14-869f-fe837ec83037-kube-api-access-tn2gn\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.075554 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074953    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-multus-conf-dir\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.075554 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.074981    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.075554 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.075007    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49ad95da-9ff8-462b-86aa-6f6eedf9ae3e-host\") pod \"node-ca-q9tbf\" (UID: \"49ad95da-9ff8-462b-86aa-6f6eedf9ae3e\") " pod="openshift-image-registry/node-ca-q9tbf"
May 28 07:59:09.075554 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.075030    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-multus-daemon-config\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.075554 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.075054    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggmwq\" (UniqueName: \"kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq\") pod \"network-check-target-t782k\" (UID: \"b33f26fc-296c-4693-b53a-a88ccd6f2236\") " pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:09.075554 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.075075    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-cnibin\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.075554 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.075112    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-cni-binary-copy\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.075554 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.075137    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64ffc18e-c32d-4d14-869f-fe837ec83037-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.075554 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.075157    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/520071c5-2f56-457c-9fc5-d833415d3248-ovnkube-script-lib\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.075554 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.075178    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64ffc18e-c32d-4d14-869f-fe837ec83037-system-cni-dir\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.076544 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.076527    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 28 07:59:09.076639 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.076606    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 28 07:59:09.076639 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.076613    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mwvmm\""
May 28 07:59:09.077183 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.077168    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8vcnc"
May 28 07:59:09.078629 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.078613    2551 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 28 07:59:09.079293 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.079278    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 28 07:59:09.079394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.079366    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 28 07:59:09.079490 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.079477    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hhqkm\""
May 28 07:59:09.096711 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.096689    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9fbj5"
May 28 07:59:09.104188 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.104169    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9fbj5"
May 28 07:59:09.143023 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:09.142984    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0a0cad24c1a6672f2575bc0047f126.slice/crio-8a94c8b9debaf6f80ba6225e3fdfc8774e3d0d481b7739a8dd43ba735aa1910e WatchSource:0}: Error finding container 8a94c8b9debaf6f80ba6225e3fdfc8774e3d0d481b7739a8dd43ba735aa1910e: Status 404 returned error can't find the container with id 8a94c8b9debaf6f80ba6225e3fdfc8774e3d0d481b7739a8dd43ba735aa1910e
May 28 07:59:09.143251 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:09.143224    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5062c11c04938e57cbece55012367eff.slice/crio-b5cb2343fb62c5c51a9d93215bc23cf5cb02fef774a88e034ec69deccfd4950c WatchSource:0}: Error finding container b5cb2343fb62c5c51a9d93215bc23cf5cb02fef774a88e034ec69deccfd4950c: Status 404 returned error can't find the container with id b5cb2343fb62c5c51a9d93215bc23cf5cb02fef774a88e034ec69deccfd4950c
May 28 07:59:09.147624 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.147607    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 07:59:09.168974 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.168953    2551 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 28 07:59:09.175666 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.175644    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-var-lib-cni-multus\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.175786 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.175672    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a310a270-ab14-4bfc-ade4-123b9ee11db4-agent-certs\") pod \"konnectivity-agent-nfsn2\" (UID: \"a310a270-ab14-4bfc-ade4-123b9ee11db4\") " pod="kube-system/konnectivity-agent-nfsn2"
May 28 07:59:09.175786 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.175687    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-log-socket\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.175786 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.175717    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-run-ovn-kubernetes\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.175786 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.175743    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9-iptables-alerter-script\") pod \"iptables-alerter-p6zng\" (UID: \"1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9\") " pod="openshift-network-operator/iptables-alerter-p6zng"
May 28 07:59:09.175786 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.175750    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-var-lib-cni-multus\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.175786 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.175771    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-var-lib-kubelet\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.176054 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.175793    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-run-ovn-kubernetes\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.176054 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.175756    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-log-socket\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.176054 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.175803    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-var-lib-cni-bin\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.176054 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.175855    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-var-lib-cni-bin\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.176054 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.175901    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49ad95da-9ff8-462b-86aa-6f6eedf9ae3e-serviceca\") pod \"node-ca-q9tbf\" (UID: \"49ad95da-9ff8-462b-86aa-6f6eedf9ae3e\") " pod="openshift-image-registry/node-ca-q9tbf"
May 28 07:59:09.176054 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.175948    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-etc-kubernetes\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.176054 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.175975    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-etc-openvswitch\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.176054 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.175998    2551 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 28 07:59:09.176054 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176044    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-run-netns\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.176054 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176023    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-etc-kubernetes\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176003    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-run-netns\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176077    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-etc-openvswitch\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176091    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-var-lib-kubelet\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176126    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-var-lib-kubelet\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176133    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-kubelet\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176158    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-run-netns\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176184    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-node-log\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176229    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9-host-slash\") pod \"iptables-alerter-p6zng\" (UID: \"1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9\") " pod="openshift-network-operator/iptables-alerter-p6zng"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176249    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dl7dc\" (UniqueName: \"kubernetes.io/projected/49ad95da-9ff8-462b-86aa-6f6eedf9ae3e-kube-api-access-dl7dc\") pod \"node-ca-q9tbf\" (UID: \"49ad95da-9ff8-462b-86aa-6f6eedf9ae3e\") " pod="openshift-image-registry/node-ca-q9tbf"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176279    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs\") pod \"network-metrics-daemon-dfswh\" (UID: \"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd\") " pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176297    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9-iptables-alerter-script\") pod \"iptables-alerter-p6zng\" (UID: \"1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9\") " pod="openshift-network-operator/iptables-alerter-p6zng"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176306    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-modprobe-d\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176322    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9-host-slash\") pod \"iptables-alerter-p6zng\" (UID: \"1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9\") " pod="openshift-network-operator/iptables-alerter-p6zng"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176334    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-run-systemd\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176307    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-kubelet\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176371    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-node-log\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.176511 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176410    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vpxg8\" (UniqueName: \"kubernetes.io/projected/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-kube-api-access-vpxg8\") pod \"network-metrics-daemon-dfswh\" (UID: \"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd\") " pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176429    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-run-systemd\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176334    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-run-netns\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176437    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tdtts\" (UniqueName: \"kubernetes.io/projected/6fcecffb-0699-457c-a17f-be7801546dd8-kube-api-access-tdtts\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:09.176464    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176482    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-host\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176428    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49ad95da-9ff8-462b-86aa-6f6eedf9ae3e-serviceca\") pod \"node-ca-q9tbf\" (UID: \"49ad95da-9ff8-462b-86aa-6f6eedf9ae3e\") " pod="openshift-image-registry/node-ca-q9tbf"
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:09.176540    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs podName:2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd nodeName:}" failed. No retries permitted until 2026-05-28 07:59:09.67651163 +0000 UTC m=+2.116583903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs") pod "network-metrics-daemon-dfswh" (UID: "2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176562    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-tuned\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176589    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-multus-cni-dir\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176616    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-run-multus-certs\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176643    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9brpl\" (UniqueName: \"kubernetes.io/projected/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-kube-api-access-9brpl\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176667    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-slash\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176672    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-run-multus-certs\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176677    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-multus-cni-dir\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176691    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64ffc18e-c32d-4d14-869f-fe837ec83037-os-release\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176719    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-sysctl-d\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.177263 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176725    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-slash\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176738    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64ffc18e-c32d-4d14-869f-fe837ec83037-os-release\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176739    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-systemd-units\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176769    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-var-lib-openvswitch\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176794    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-hostroot\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176818    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/520071c5-2f56-457c-9fc5-d833415d3248-env-overrides\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176810    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-systemd-units\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176882    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-cni-bin\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176874    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-hostroot\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176897    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-var-lib-openvswitch\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176926    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-cni-bin\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176930    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vphsz\" (UniqueName: \"kubernetes.io/projected/520071c5-2f56-457c-9fc5-d833415d3248-kube-api-access-vphsz\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.176953    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rn7xf\" (UniqueName: \"kubernetes.io/projected/1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9-kube-api-access-rn7xf\") pod \"iptables-alerter-p6zng\" (UID: \"1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9\") " pod="openshift-network-operator/iptables-alerter-p6zng"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177083    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-socket-dir\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177099    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177114    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64ffc18e-c32d-4d14-869f-fe837ec83037-cni-binary-copy\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177183    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.178014 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177203    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177215    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-device-dir\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177240    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64ffc18e-c32d-4d14-869f-fe837ec83037-cnibin\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177255    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-device-dir\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177255    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177267    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64ffc18e-c32d-4d14-869f-fe837ec83037-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177287    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64ffc18e-c32d-4d14-869f-fe837ec83037-cnibin\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177217    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-socket-dir\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177299    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64ffc18e-c32d-4d14-869f-fe837ec83037-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177328    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91168ae0-b0ae-482d-be58-f879dcae55f1-tmp-dir\") pod \"node-resolver-8vcnc\" (UID: \"91168ae0-b0ae-482d-be58-f879dcae55f1\") " pod="openshift-dns/node-resolver-8vcnc"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177356    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tn2gn\" (UniqueName: \"kubernetes.io/projected/64ffc18e-c32d-4d14-869f-fe837ec83037-kube-api-access-tn2gn\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177349    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/520071c5-2f56-457c-9fc5-d833415d3248-env-overrides\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177404    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91168ae0-b0ae-482d-be58-f879dcae55f1-hosts-file\") pod \"node-resolver-8vcnc\" (UID: \"91168ae0-b0ae-482d-be58-f879dcae55f1\") " pod="openshift-dns/node-resolver-8vcnc"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177423    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64ffc18e-c32d-4d14-869f-fe837ec83037-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177431    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-multus-conf-dir\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177461    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177489    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49ad95da-9ff8-462b-86aa-6f6eedf9ae3e-host\") pod \"node-ca-q9tbf\" (UID: \"49ad95da-9ff8-462b-86aa-6f6eedf9ae3e\") " pod="openshift-image-registry/node-ca-q9tbf"
May 28 07:59:09.178780 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177515    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-multus-daemon-config\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177495    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-multus-conf-dir\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177545    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmwq\" (UniqueName: \"kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq\") pod \"network-check-target-t782k\" (UID: \"b33f26fc-296c-4693-b53a-a88ccd6f2236\") " pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177548    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177569    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49ad95da-9ff8-462b-86aa-6f6eedf9ae3e-host\") pod \"node-ca-q9tbf\" (UID: \"49ad95da-9ff8-462b-86aa-6f6eedf9ae3e\") " pod="openshift-image-registry/node-ca-q9tbf"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177571    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-cnibin\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177621    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-cnibin\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177656    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-cni-binary-copy\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177684    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64ffc18e-c32d-4d14-869f-fe837ec83037-cni-binary-copy\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177716    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64ffc18e-c32d-4d14-869f-fe837ec83037-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177758    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/520071c5-2f56-457c-9fc5-d833415d3248-ovnkube-script-lib\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177783    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64ffc18e-c32d-4d14-869f-fe837ec83037-system-cni-dir\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177847    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-lib-modules\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177862    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64ffc18e-c32d-4d14-869f-fe837ec83037-system-cni-dir\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177811    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64ffc18e-c32d-4d14-869f-fe837ec83037-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177895    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvnqt\" (UniqueName: \"kubernetes.io/projected/91168ae0-b0ae-482d-be58-f879dcae55f1-kube-api-access-jvnqt\") pod \"node-resolver-8vcnc\" (UID: \"91168ae0-b0ae-482d-be58-f879dcae55f1\") " pod="openshift-dns/node-resolver-8vcnc"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177939    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-multus-socket-dir-parent\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.179394 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177963    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-run-ovn\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.177990    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-cni-netd\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178025    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-multus-socket-dir-parent\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178015    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a310a270-ab14-4bfc-ade4-123b9ee11db4-konnectivity-ca\") pod \"konnectivity-agent-nfsn2\" (UID: \"a310a270-ab14-4bfc-ade4-123b9ee11db4\") " pod="kube-system/konnectivity-agent-nfsn2"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178073    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-systemd\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178099    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-tmp\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178103    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-multus-daemon-config\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178124    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cflt2\" (UniqueName: \"kubernetes.io/projected/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-kube-api-access-cflt2\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178136    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-cni-binary-copy\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178155    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-run-k8s-cni-cncf-io\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178163    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-run-ovn\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178180    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/520071c5-2f56-457c-9fc5-d833415d3248-ovn-node-metrics-cert\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178210    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-host-run-k8s-cni-cncf-io\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178218    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64ffc18e-c32d-4d14-869f-fe837ec83037-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178232    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-sys-fs\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178213    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-host-cni-netd\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178260    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-sysconfig\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.179866 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178303    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-kubernetes\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178327    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-sys-fs\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178328    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/520071c5-2f56-457c-9fc5-d833415d3248-ovnkube-script-lib\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178329    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-sysctl-conf\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178360    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-sys\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178404    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-system-cni-dir\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178450    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-system-cni-dir\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178491    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a310a270-ab14-4bfc-ade4-123b9ee11db4-konnectivity-ca\") pod \"konnectivity-agent-nfsn2\" (UID: \"a310a270-ab14-4bfc-ade4-123b9ee11db4\") " pod="kube-system/konnectivity-agent-nfsn2"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178506    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-run-openvswitch\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178533    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/520071c5-2f56-457c-9fc5-d833415d3248-ovnkube-config\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178559    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-registration-dir\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178553    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/520071c5-2f56-457c-9fc5-d833415d3248-run-openvswitch\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178585    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-os-release\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178613    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-run\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178682    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-os-release\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.178682    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6fcecffb-0699-457c-a17f-be7801546dd8-registration-dir\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.179000    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/520071c5-2f56-457c-9fc5-d833415d3248-ovnkube-config\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.179485    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a310a270-ab14-4bfc-ade4-123b9ee11db4-agent-certs\") pod \"konnectivity-agent-nfsn2\" (UID: \"a310a270-ab14-4bfc-ade4-123b9ee11db4\") " pod="kube-system/konnectivity-agent-nfsn2"
May 28 07:59:09.180313 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.180121    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/520071c5-2f56-457c-9fc5-d833415d3248-ovn-node-metrics-cert\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.183690 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:09.183445    2551 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 07:59:09.183690 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:09.183470    2551 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 07:59:09.183690 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:09.183483    2551 projected.go:194] Error preparing data for projected volume kube-api-access-ggmwq for pod openshift-network-diagnostics/network-check-target-t782k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 07:59:09.183690 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:09.183541    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq podName:b33f26fc-296c-4693-b53a-a88ccd6f2236 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:09.683523119 +0000 UTC m=+2.123595403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ggmwq" (UniqueName: "kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq") pod "network-check-target-t782k" (UID: "b33f26fc-296c-4693-b53a-a88ccd6f2236") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 07:59:09.185030 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.184998    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl7dc\" (UniqueName: \"kubernetes.io/projected/49ad95da-9ff8-462b-86aa-6f6eedf9ae3e-kube-api-access-dl7dc\") pod \"node-ca-q9tbf\" (UID: \"49ad95da-9ff8-462b-86aa-6f6eedf9ae3e\") " pod="openshift-image-registry/node-ca-q9tbf"
May 28 07:59:09.185343 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.185326    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpxg8\" (UniqueName: \"kubernetes.io/projected/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-kube-api-access-vpxg8\") pod \"network-metrics-daemon-dfswh\" (UID: \"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd\") " pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:09.185343 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.185331    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdtts\" (UniqueName: \"kubernetes.io/projected/6fcecffb-0699-457c-a17f-be7801546dd8-kube-api-access-tdtts\") pod \"aws-ebs-csi-driver-node-mq2lj\" (UID: \"6fcecffb-0699-457c-a17f-be7801546dd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.185472 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.185316    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vphsz\" (UniqueName: \"kubernetes.io/projected/520071c5-2f56-457c-9fc5-d833415d3248-kube-api-access-vphsz\") pod \"ovnkube-node-vvpbp\" (UID: \"520071c5-2f56-457c-9fc5-d833415d3248\") " pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.185472 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.185368    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brpl\" (UniqueName: \"kubernetes.io/projected/7bce0101-b6a3-4da0-bab9-10edb7bcad0b-kube-api-access-9brpl\") pod \"multus-fjgcp\" (UID: \"7bce0101-b6a3-4da0-bab9-10edb7bcad0b\") " pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.185806 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.185789    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn7xf\" (UniqueName: \"kubernetes.io/projected/1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9-kube-api-access-rn7xf\") pod \"iptables-alerter-p6zng\" (UID: \"1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9\") " pod="openshift-network-operator/iptables-alerter-p6zng"
May 28 07:59:09.185857 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.185816    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn2gn\" (UniqueName: \"kubernetes.io/projected/64ffc18e-c32d-4d14-869f-fe837ec83037-kube-api-access-tn2gn\") pod \"multus-additional-cni-plugins-wcbtq\" (UID: \"64ffc18e-c32d-4d14-869f-fe837ec83037\") " pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.193512 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.193474    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-248.ec2.internal" event={"ID":"fc0a0cad24c1a6672f2575bc0047f126","Type":"ContainerStarted","Data":"8a94c8b9debaf6f80ba6225e3fdfc8774e3d0d481b7739a8dd43ba735aa1910e"}
May 28 07:59:09.194406 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.194369    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal" event={"ID":"5062c11c04938e57cbece55012367eff","Type":"ContainerStarted","Data":"b5cb2343fb62c5c51a9d93215bc23cf5cb02fef774a88e034ec69deccfd4950c"}
May 28 07:59:09.279233 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279144    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-sysctl-d\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279233 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279181    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91168ae0-b0ae-482d-be58-f879dcae55f1-tmp-dir\") pod \"node-resolver-8vcnc\" (UID: \"91168ae0-b0ae-482d-be58-f879dcae55f1\") " pod="openshift-dns/node-resolver-8vcnc"
May 28 07:59:09.279233 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279197    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91168ae0-b0ae-482d-be58-f879dcae55f1-hosts-file\") pod \"node-resolver-8vcnc\" (UID: \"91168ae0-b0ae-482d-be58-f879dcae55f1\") " pod="openshift-dns/node-resolver-8vcnc"
May 28 07:59:09.279233 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279226    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-lib-modules\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279241    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jvnqt\" (UniqueName: \"kubernetes.io/projected/91168ae0-b0ae-482d-be58-f879dcae55f1-kube-api-access-jvnqt\") pod \"node-resolver-8vcnc\" (UID: \"91168ae0-b0ae-482d-be58-f879dcae55f1\") " pod="openshift-dns/node-resolver-8vcnc"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279264    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-systemd\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279282    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-tmp\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279300    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cflt2\" (UniqueName: \"kubernetes.io/projected/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-kube-api-access-cflt2\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279317    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-sysconfig\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279316    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91168ae0-b0ae-482d-be58-f879dcae55f1-hosts-file\") pod \"node-resolver-8vcnc\" (UID: \"91168ae0-b0ae-482d-be58-f879dcae55f1\") " pod="openshift-dns/node-resolver-8vcnc"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279334    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-kubernetes\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279350    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-sysctl-d\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279363    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-lib-modules\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279373    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-systemd\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279379    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-sysctl-conf\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279443    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-sys\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279464    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-run\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279490    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-var-lib-kubelet\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279509    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-sysctl-conf\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279520    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-sys\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279549    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-modprobe-d\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.279581 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279554    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91168ae0-b0ae-482d-be58-f879dcae55f1-tmp-dir\") pod \"node-resolver-8vcnc\" (UID: \"91168ae0-b0ae-482d-be58-f879dcae55f1\") " pod="openshift-dns/node-resolver-8vcnc"
May 28 07:59:09.280140 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279562    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-run\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.280140 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279405    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-kubernetes\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.280140 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279569    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-var-lib-kubelet\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.280140 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279622    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-host\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.280140 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279631    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-sysconfig\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.280140 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279642    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-modprobe-d\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.280140 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279648    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-tuned\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.280140 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.279675    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-host\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.281453 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.281435    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-tmp\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.281527 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.281465    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-etc-tuned\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.287526 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.287503    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cflt2\" (UniqueName: \"kubernetes.io/projected/ae6d6f2a-72d0-44f2-b512-e739ccf792eb-kube-api-access-cflt2\") pod \"tuned-lfc7s\" (UID: \"ae6d6f2a-72d0-44f2-b512-e739ccf792eb\") " pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.287604 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.287513    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvnqt\" (UniqueName: \"kubernetes.io/projected/91168ae0-b0ae-482d-be58-f879dcae55f1-kube-api-access-jvnqt\") pod \"node-resolver-8vcnc\" (UID: \"91168ae0-b0ae-482d-be58-f879dcae55f1\") " pod="openshift-dns/node-resolver-8vcnc"
May 28 07:59:09.381585 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.381556    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wcbtq"
May 28 07:59:09.388554 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:09.388521    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ffc18e_c32d_4d14_869f_fe837ec83037.slice/crio-99ca848c46493c6ae20f2fc32510c2b35e976bb6c11716c878ba6dda3dacf956 WatchSource:0}: Error finding container 99ca848c46493c6ae20f2fc32510c2b35e976bb6c11716c878ba6dda3dacf956: Status 404 returned error can't find the container with id 99ca848c46493c6ae20f2fc32510c2b35e976bb6c11716c878ba6dda3dacf956
May 28 07:59:09.401686 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.401662    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p6zng"
May 28 07:59:09.408716 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:09.408692    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe5d6a6_0c2f_4e92_a246_e92a2f6f14a9.slice/crio-39e552b523476c769a7c80d04e459ab89c2cd834f0522ee882a635418a9365e5 WatchSource:0}: Error finding container 39e552b523476c769a7c80d04e459ab89c2cd834f0522ee882a635418a9365e5: Status 404 returned error can't find the container with id 39e552b523476c769a7c80d04e459ab89c2cd834f0522ee882a635418a9365e5
May 28 07:59:09.428148 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.428128    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q9tbf"
May 28 07:59:09.434361 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:09.434336    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ad95da_9ff8_462b_86aa_6f6eedf9ae3e.slice/crio-675093046b3a2935c28cdf03500c96ec77a00e0f06936f71259c805ff22a67a2 WatchSource:0}: Error finding container 675093046b3a2935c28cdf03500c96ec77a00e0f06936f71259c805ff22a67a2: Status 404 returned error can't find the container with id 675093046b3a2935c28cdf03500c96ec77a00e0f06936f71259c805ff22a67a2
May 28 07:59:09.444717 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.444696    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fjgcp"
May 28 07:59:09.450209 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.450174    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:09.450450 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:09.450427    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bce0101_b6a3_4da0_bab9_10edb7bcad0b.slice/crio-f89f58cf7aaa301239a8b744f3a10d8089a5f297ba373b9f78d635ad08762541 WatchSource:0}: Error finding container f89f58cf7aaa301239a8b744f3a10d8089a5f297ba373b9f78d635ad08762541: Status 404 returned error can't find the container with id f89f58cf7aaa301239a8b744f3a10d8089a5f297ba373b9f78d635ad08762541
May 28 07:59:09.455936 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.455912    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nfsn2"
May 28 07:59:09.458105 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:09.458077    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520071c5_2f56_457c_9fc5_d833415d3248.slice/crio-b502f234338a7748ba473324a26931ebec59810acdc1503041e8d1d0b7d901f0 WatchSource:0}: Error finding container b502f234338a7748ba473324a26931ebec59810acdc1503041e8d1d0b7d901f0: Status 404 returned error can't find the container with id b502f234338a7748ba473324a26931ebec59810acdc1503041e8d1d0b7d901f0
May 28 07:59:09.461196 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.461174    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj"
May 28 07:59:09.466422 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:09.466376    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda310a270_ab14_4bfc_ade4_123b9ee11db4.slice/crio-160f9092a4551dad9b06fb7e9fadfb47a8b6b8dab259418d12cfae2cc5feebfe WatchSource:0}: Error finding container 160f9092a4551dad9b06fb7e9fadfb47a8b6b8dab259418d12cfae2cc5feebfe: Status 404 returned error can't find the container with id 160f9092a4551dad9b06fb7e9fadfb47a8b6b8dab259418d12cfae2cc5feebfe
May 28 07:59:09.467352 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.467332    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lfc7s"
May 28 07:59:09.469785 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:09.469760    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcecffb_0699_457c_a17f_be7801546dd8.slice/crio-1842568607a38f26699f9f34362e6c8614cff3d19da0482a383ff9fdb43a033b WatchSource:0}: Error finding container 1842568607a38f26699f9f34362e6c8614cff3d19da0482a383ff9fdb43a033b: Status 404 returned error can't find the container with id 1842568607a38f26699f9f34362e6c8614cff3d19da0482a383ff9fdb43a033b
May 28 07:59:09.471113 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.471093    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8vcnc"
May 28 07:59:09.477557 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:09.477533    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6d6f2a_72d0_44f2_b512_e739ccf792eb.slice/crio-50ad6b56db13d90b22e45f3c4472d4aebd59684afe996ee672efb4d62a021da4 WatchSource:0}: Error finding container 50ad6b56db13d90b22e45f3c4472d4aebd59684afe996ee672efb4d62a021da4: Status 404 returned error can't find the container with id 50ad6b56db13d90b22e45f3c4472d4aebd59684afe996ee672efb4d62a021da4
May 28 07:59:09.484480 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:09.481804    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91168ae0_b0ae_482d_be58_f879dcae55f1.slice/crio-2b772c86ed480db67cbde2624a6a4a3c9e23830f6c550006f9dd9dbaa5036eb3 WatchSource:0}: Error finding container 2b772c86ed480db67cbde2624a6a4a3c9e23830f6c550006f9dd9dbaa5036eb3: Status 404 returned error can't find the container with id 2b772c86ed480db67cbde2624a6a4a3c9e23830f6c550006f9dd9dbaa5036eb3
May 28 07:59:09.683046 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.682966    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs\") pod \"network-metrics-daemon-dfswh\" (UID: \"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd\") " pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:09.683209 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:09.683139    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 07:59:09.683269 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:09.683208    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs podName:2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd nodeName:}" failed. No retries permitted until 2026-05-28 07:59:10.683188627 +0000 UTC m=+3.123260911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs") pod "network-metrics-daemon-dfswh" (UID: "2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 07:59:09.783964 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.783594    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmwq\" (UniqueName: \"kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq\") pod \"network-check-target-t782k\" (UID: \"b33f26fc-296c-4693-b53a-a88ccd6f2236\") " pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:09.783964 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:09.783936    2551 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 07:59:09.783964 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:09.783967    2551 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 07:59:09.784258 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:09.783981    2551 projected.go:194] Error preparing data for projected volume kube-api-access-ggmwq for pod openshift-network-diagnostics/network-check-target-t782k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 07:59:09.784258 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:09.784105    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq podName:b33f26fc-296c-4693-b53a-a88ccd6f2236 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:10.784085209 +0000 UTC m=+3.224157479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ggmwq" (UniqueName: "kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq") pod "network-check-target-t782k" (UID: "b33f26fc-296c-4693-b53a-a88ccd6f2236") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 07:59:09.943842 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:09.943767    2551 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 07:59:10.105110 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.105071    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 07:54:09 +0000 UTC" deadline="2027-11-06 06:20:43.533974539 +0000 UTC"
May 28 07:59:10.105110 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.105104    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12646h21m33.428872297s"
May 28 07:59:10.193060 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.193026    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:10.193232 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:10.193158    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:10.211976 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.210939    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8vcnc" event={"ID":"91168ae0-b0ae-482d-be58-f879dcae55f1","Type":"ContainerStarted","Data":"2b772c86ed480db67cbde2624a6a4a3c9e23830f6c550006f9dd9dbaa5036eb3"}
May 28 07:59:10.223574 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.223538    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nfsn2" event={"ID":"a310a270-ab14-4bfc-ade4-123b9ee11db4","Type":"ContainerStarted","Data":"160f9092a4551dad9b06fb7e9fadfb47a8b6b8dab259418d12cfae2cc5feebfe"}
May 28 07:59:10.234266 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.234232    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q9tbf" event={"ID":"49ad95da-9ff8-462b-86aa-6f6eedf9ae3e","Type":"ContainerStarted","Data":"675093046b3a2935c28cdf03500c96ec77a00e0f06936f71259c805ff22a67a2"}
May 28 07:59:10.238489 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.238462    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcbtq" event={"ID":"64ffc18e-c32d-4d14-869f-fe837ec83037","Type":"ContainerStarted","Data":"99ca848c46493c6ae20f2fc32510c2b35e976bb6c11716c878ba6dda3dacf956"}
May 28 07:59:10.241630 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.241605    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lfc7s" event={"ID":"ae6d6f2a-72d0-44f2-b512-e739ccf792eb","Type":"ContainerStarted","Data":"50ad6b56db13d90b22e45f3c4472d4aebd59684afe996ee672efb4d62a021da4"}
May 28 07:59:10.247556 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.247530    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj" event={"ID":"6fcecffb-0699-457c-a17f-be7801546dd8","Type":"ContainerStarted","Data":"1842568607a38f26699f9f34362e6c8614cff3d19da0482a383ff9fdb43a033b"}
May 28 07:59:10.254524 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.254475    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp" event={"ID":"520071c5-2f56-457c-9fc5-d833415d3248","Type":"ContainerStarted","Data":"b502f234338a7748ba473324a26931ebec59810acdc1503041e8d1d0b7d901f0"}
May 28 07:59:10.259152 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.259127    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fjgcp" event={"ID":"7bce0101-b6a3-4da0-bab9-10edb7bcad0b","Type":"ContainerStarted","Data":"f89f58cf7aaa301239a8b744f3a10d8089a5f297ba373b9f78d635ad08762541"}
May 28 07:59:10.271677 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.271651    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p6zng" event={"ID":"1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9","Type":"ContainerStarted","Data":"39e552b523476c769a7c80d04e459ab89c2cd834f0522ee882a635418a9365e5"}
May 28 07:59:10.391078 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.391047    2551 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 07:59:10.467623 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.467530    2551 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 28 07:59:10.697235 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.697195    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs\") pod \"network-metrics-daemon-dfswh\" (UID: \"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd\") " pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:10.697482 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:10.697464    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 07:59:10.697543 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:10.697534    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs podName:2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd nodeName:}" failed. No retries permitted until 2026-05-28 07:59:12.697515151 +0000 UTC m=+5.137587436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs") pod "network-metrics-daemon-dfswh" (UID: "2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 07:59:10.798540 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:10.798461    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmwq\" (UniqueName: \"kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq\") pod \"network-check-target-t782k\" (UID: \"b33f26fc-296c-4693-b53a-a88ccd6f2236\") " pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:10.798702 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:10.798647    2551 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 07:59:10.798702 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:10.798667    2551 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 07:59:10.798702 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:10.798679    2551 projected.go:194] Error preparing data for projected volume kube-api-access-ggmwq for pod openshift-network-diagnostics/network-check-target-t782k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 07:59:10.798851 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:10.798740    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq podName:b33f26fc-296c-4693-b53a-a88ccd6f2236 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:12.798718408 +0000 UTC m=+5.238790695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ggmwq" (UniqueName: "kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq") pod "network-check-target-t782k" (UID: "b33f26fc-296c-4693-b53a-a88ccd6f2236") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 07:59:11.105560 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:11.105475    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 07:54:09 +0000 UTC" deadline="2027-11-22 21:52:02.810017235 +0000 UTC"
May 28 07:59:11.105560 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:11.105518    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13045h52m51.704503751s"
May 28 07:59:11.190845 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:11.190813    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:11.191009 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:11.190961    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:12.191498 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:12.191190    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:12.191498 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:12.191342    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:12.715010 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:12.714971    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs\") pod \"network-metrics-daemon-dfswh\" (UID: \"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd\") " pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:12.715196 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:12.715159    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 07:59:12.715254 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:12.715222    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs podName:2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd nodeName:}" failed. No retries permitted until 2026-05-28 07:59:16.71520278 +0000 UTC m=+9.155275053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs") pod "network-metrics-daemon-dfswh" (UID: "2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 07:59:12.815603 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:12.815551    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmwq\" (UniqueName: \"kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq\") pod \"network-check-target-t782k\" (UID: \"b33f26fc-296c-4693-b53a-a88ccd6f2236\") " pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:12.815772 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:12.815738    2551 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 07:59:12.815772 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:12.815757    2551 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 07:59:12.815772 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:12.815769    2551 projected.go:194] Error preparing data for projected volume kube-api-access-ggmwq for pod openshift-network-diagnostics/network-check-target-t782k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 07:59:12.815937 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:12.815831    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq podName:b33f26fc-296c-4693-b53a-a88ccd6f2236 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:16.815813604 +0000 UTC m=+9.255885877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ggmwq" (UniqueName: "kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq") pod "network-check-target-t782k" (UID: "b33f26fc-296c-4693-b53a-a88ccd6f2236") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 07:59:13.191011 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:13.190930    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:13.191162 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:13.191066    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:14.191233 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:14.191196    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:14.191754 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:14.191325    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:15.191107 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:15.191077    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:15.191290 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:15.191193    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:16.191088 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:16.191056    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:16.191253 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:16.191186    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:16.744864 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:16.744822    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs\") pod \"network-metrics-daemon-dfswh\" (UID: \"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd\") " pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:16.745320 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:16.744952    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 07:59:16.745320 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:16.745033    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs podName:2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd nodeName:}" failed. No retries permitted until 2026-05-28 07:59:24.745010508 +0000 UTC m=+17.185082781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs") pod "network-metrics-daemon-dfswh" (UID: "2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 07:59:16.846032 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:16.845947    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmwq\" (UniqueName: \"kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq\") pod \"network-check-target-t782k\" (UID: \"b33f26fc-296c-4693-b53a-a88ccd6f2236\") " pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:16.846225 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:16.846115    2551 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 07:59:16.846225 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:16.846135    2551 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 07:59:16.846225 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:16.846149    2551 projected.go:194] Error preparing data for projected volume kube-api-access-ggmwq for pod openshift-network-diagnostics/network-check-target-t782k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 07:59:16.846225 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:16.846218    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq podName:b33f26fc-296c-4693-b53a-a88ccd6f2236 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:24.846197561 +0000 UTC m=+17.286269846 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ggmwq" (UniqueName: "kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq") pod "network-check-target-t782k" (UID: "b33f26fc-296c-4693-b53a-a88ccd6f2236") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 07:59:17.190396 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:17.190304    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:17.190545 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:17.190473    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:18.194341 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:18.194311    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:18.194783 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:18.194455    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:19.190686 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:19.190655    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:19.190873 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:19.190792    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:20.190739 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:20.190645    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:20.191281 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:20.190796    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:20.906055 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:20.906023    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-cnjvq"]
May 28 07:59:20.923892 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:20.923856    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:20.924043 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:20.923949    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cnjvq" podUID="a659431e-376b-4589-8ae4-27694a9b763f"
May 28 07:59:20.978474 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:20.978442    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret\") pod \"global-pull-secret-syncer-cnjvq\" (UID: \"a659431e-376b-4589-8ae4-27694a9b763f\") " pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:20.978629 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:20.978482    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a659431e-376b-4589-8ae4-27694a9b763f-dbus\") pod \"global-pull-secret-syncer-cnjvq\" (UID: \"a659431e-376b-4589-8ae4-27694a9b763f\") " pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:20.978629 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:20.978576    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a659431e-376b-4589-8ae4-27694a9b763f-kubelet-config\") pod \"global-pull-secret-syncer-cnjvq\" (UID: \"a659431e-376b-4589-8ae4-27694a9b763f\") " pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:21.079329 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:21.079285    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a659431e-376b-4589-8ae4-27694a9b763f-kubelet-config\") pod \"global-pull-secret-syncer-cnjvq\" (UID: \"a659431e-376b-4589-8ae4-27694a9b763f\") " pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:21.079521 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:21.079353    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret\") pod \"global-pull-secret-syncer-cnjvq\" (UID: \"a659431e-376b-4589-8ae4-27694a9b763f\") " pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:21.079521 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:21.079399    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a659431e-376b-4589-8ae4-27694a9b763f-dbus\") pod \"global-pull-secret-syncer-cnjvq\" (UID: \"a659431e-376b-4589-8ae4-27694a9b763f\") " pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:21.079521 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:21.079473    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a659431e-376b-4589-8ae4-27694a9b763f-kubelet-config\") pod \"global-pull-secret-syncer-cnjvq\" (UID: \"a659431e-376b-4589-8ae4-27694a9b763f\") " pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:21.079521 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:21.079498    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 07:59:21.079721 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:21.079561    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret podName:a659431e-376b-4589-8ae4-27694a9b763f nodeName:}" failed. No retries permitted until 2026-05-28 07:59:21.579543105 +0000 UTC m=+14.019615375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret") pod "global-pull-secret-syncer-cnjvq" (UID: "a659431e-376b-4589-8ae4-27694a9b763f") : object "kube-system"/"original-pull-secret" not registered
May 28 07:59:21.079721 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:21.079578    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a659431e-376b-4589-8ae4-27694a9b763f-dbus\") pod \"global-pull-secret-syncer-cnjvq\" (UID: \"a659431e-376b-4589-8ae4-27694a9b763f\") " pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:21.190718 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:21.190636    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:21.190868 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:21.190778    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:21.584319 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:21.584277    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret\") pod \"global-pull-secret-syncer-cnjvq\" (UID: \"a659431e-376b-4589-8ae4-27694a9b763f\") " pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:21.584493 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:21.584471    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 07:59:21.584569 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:21.584551    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret podName:a659431e-376b-4589-8ae4-27694a9b763f nodeName:}" failed. No retries permitted until 2026-05-28 07:59:22.58453448 +0000 UTC m=+15.024606762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret") pod "global-pull-secret-syncer-cnjvq" (UID: "a659431e-376b-4589-8ae4-27694a9b763f") : object "kube-system"/"original-pull-secret" not registered
May 28 07:59:22.191107 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:22.191071    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:22.191551 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:22.191202    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:22.191551 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:22.191294    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:22.191551 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:22.191430    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cnjvq" podUID="a659431e-376b-4589-8ae4-27694a9b763f"
May 28 07:59:22.592016 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:22.591982    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret\") pod \"global-pull-secret-syncer-cnjvq\" (UID: \"a659431e-376b-4589-8ae4-27694a9b763f\") " pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:22.592199 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:22.592103    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 07:59:22.592199 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:22.592168    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret podName:a659431e-376b-4589-8ae4-27694a9b763f nodeName:}" failed. No retries permitted until 2026-05-28 07:59:24.592149067 +0000 UTC m=+17.032221357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret") pod "global-pull-secret-syncer-cnjvq" (UID: "a659431e-376b-4589-8ae4-27694a9b763f") : object "kube-system"/"original-pull-secret" not registered
May 28 07:59:23.191075 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:23.191043    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:23.191253 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:23.191154    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:24.191215 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:24.191178    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:24.191379 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:24.191178    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:24.191379 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:24.191304    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cnjvq" podUID="a659431e-376b-4589-8ae4-27694a9b763f"
May 28 07:59:24.191379 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:24.191348    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:24.608566 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:24.608522    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret\") pod \"global-pull-secret-syncer-cnjvq\" (UID: \"a659431e-376b-4589-8ae4-27694a9b763f\") " pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:24.608786 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:24.608677    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 07:59:24.608786 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:24.608753    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret podName:a659431e-376b-4589-8ae4-27694a9b763f nodeName:}" failed. No retries permitted until 2026-05-28 07:59:28.608731068 +0000 UTC m=+21.048803352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret") pod "global-pull-secret-syncer-cnjvq" (UID: "a659431e-376b-4589-8ae4-27694a9b763f") : object "kube-system"/"original-pull-secret" not registered
May 28 07:59:24.809661 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:24.809628    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs\") pod \"network-metrics-daemon-dfswh\" (UID: \"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd\") " pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:24.809829 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:24.809790    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 07:59:24.809893 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:24.809859    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs podName:2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd nodeName:}" failed. No retries permitted until 2026-05-28 07:59:40.809833805 +0000 UTC m=+33.249906072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs") pod "network-metrics-daemon-dfswh" (UID: "2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 07:59:24.910545 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:24.910466    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmwq\" (UniqueName: \"kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq\") pod \"network-check-target-t782k\" (UID: \"b33f26fc-296c-4693-b53a-a88ccd6f2236\") " pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:24.910680 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:24.910646    2551 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 07:59:24.910680 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:24.910668    2551 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 07:59:24.910680 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:24.910681    2551 projected.go:194] Error preparing data for projected volume kube-api-access-ggmwq for pod openshift-network-diagnostics/network-check-target-t782k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 07:59:24.910815 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:24.910742    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq podName:b33f26fc-296c-4693-b53a-a88ccd6f2236 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:40.910723062 +0000 UTC m=+33.350795344 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ggmwq" (UniqueName: "kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq") pod "network-check-target-t782k" (UID: "b33f26fc-296c-4693-b53a-a88ccd6f2236") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 07:59:25.190489 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:25.190413    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:25.190647 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:25.190541    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:26.191320 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:26.191283    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:26.191765 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:26.191283    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:26.191765 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:26.191435    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cnjvq" podUID="a659431e-376b-4589-8ae4-27694a9b763f"
May 28 07:59:26.191765 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:26.191567    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:27.190656 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:27.190619    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:27.190823 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:27.190747    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:28.191419 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:28.191377    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:28.191715 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:28.191489    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cnjvq" podUID="a659431e-376b-4589-8ae4-27694a9b763f"
May 28 07:59:28.191715 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:28.191524    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:28.191715 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:28.191571    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:28.640992 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:28.640963    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret\") pod \"global-pull-secret-syncer-cnjvq\" (UID: \"a659431e-376b-4589-8ae4-27694a9b763f\") " pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:28.641118 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:28.641104    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 07:59:28.641176 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:28.641168    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret podName:a659431e-376b-4589-8ae4-27694a9b763f nodeName:}" failed. No retries permitted until 2026-05-28 07:59:36.641149077 +0000 UTC m=+29.081221364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret") pod "global-pull-secret-syncer-cnjvq" (UID: "a659431e-376b-4589-8ae4-27694a9b763f") : object "kube-system"/"original-pull-secret" not registered
May 28 07:59:29.191297 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.191270    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:29.191457 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:29.191405    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:29.309193 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.308992    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-248.ec2.internal" event={"ID":"fc0a0cad24c1a6672f2575bc0047f126","Type":"ContainerStarted","Data":"931fc9ca72eba045479d0f8aa4b092df5a4e4799387de02dd39819df10007846"}
May 28 07:59:29.310238 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.310208    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8vcnc" event={"ID":"91168ae0-b0ae-482d-be58-f879dcae55f1","Type":"ContainerStarted","Data":"b8bdb7204163cee63d89e35f439d59849a87c024b4bf0b686dfeaa292c52026d"}
May 28 07:59:29.311432 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.311408    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nfsn2" event={"ID":"a310a270-ab14-4bfc-ade4-123b9ee11db4","Type":"ContainerStarted","Data":"220916f48018a399488c114a3ff3e5e24de6cfd480cc96d2d6ad6769e7c7b8a9"}
May 28 07:59:29.312550 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.312531    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q9tbf" event={"ID":"49ad95da-9ff8-462b-86aa-6f6eedf9ae3e","Type":"ContainerStarted","Data":"0c427cbe3125dbe944942ccba7a249df81439245f1c169eab2fbf575d9732906"}
May 28 07:59:29.314093 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.314071    2551 generic.go:358] "Generic (PLEG): container finished" podID="64ffc18e-c32d-4d14-869f-fe837ec83037" containerID="dc3b11b6eb3b3029c9a8f4210068a7404659ca8994ed59d3863d7e86c61d393f" exitCode=0
May 28 07:59:29.314195 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.314131    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcbtq" event={"ID":"64ffc18e-c32d-4d14-869f-fe837ec83037","Type":"ContainerDied","Data":"dc3b11b6eb3b3029c9a8f4210068a7404659ca8994ed59d3863d7e86c61d393f"}
May 28 07:59:29.315686 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.315570    2551 generic.go:358] "Generic (PLEG): container finished" podID="5062c11c04938e57cbece55012367eff" containerID="69850e6e639ccc9cc280ec9c8cefba55df0bcae237e8dd04c39c56ce12f4747d" exitCode=0
May 28 07:59:29.315686 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.315651    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal" event={"ID":"5062c11c04938e57cbece55012367eff","Type":"ContainerDied","Data":"69850e6e639ccc9cc280ec9c8cefba55df0bcae237e8dd04c39c56ce12f4747d"}
May 28 07:59:29.317037 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.316981    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lfc7s" event={"ID":"ae6d6f2a-72d0-44f2-b512-e739ccf792eb","Type":"ContainerStarted","Data":"16ddd9b023e09eb204900e4201e73675715992d9766f3b1a88ba858838d2e87a"}
May 28 07:59:29.318343 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.318319    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj" event={"ID":"6fcecffb-0699-457c-a17f-be7801546dd8","Type":"ContainerStarted","Data":"e4c8efd45733f8ae4409e37621eb90c49b256a8ae56d942991621917ad1938bf"}
May 28 07:59:29.321022 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.320995    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp" event={"ID":"520071c5-2f56-457c-9fc5-d833415d3248","Type":"ContainerStarted","Data":"1c871a002eee3be70e547da963578bc1e279cd328d0d9c7e2129329ef19f2096"}
May 28 07:59:29.321022 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.321017    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp" event={"ID":"520071c5-2f56-457c-9fc5-d833415d3248","Type":"ContainerStarted","Data":"24c58ae36a119f4a64606fd293c76d62992722bf9bbca2e2eac48f3588b5a477"}
May 28 07:59:29.321022 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.321026    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp" event={"ID":"520071c5-2f56-457c-9fc5-d833415d3248","Type":"ContainerStarted","Data":"12b3441c793a93100f456db7b7af040550581a90c01c881d89f3d08c52ce65af"}
May 28 07:59:29.321171 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.321034    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp" event={"ID":"520071c5-2f56-457c-9fc5-d833415d3248","Type":"ContainerStarted","Data":"8d2b76afcea06808311fc5355fdf5b7ebb232ad45b33396303062d51555da5ac"}
May 28 07:59:29.321171 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.321044    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp" event={"ID":"520071c5-2f56-457c-9fc5-d833415d3248","Type":"ContainerStarted","Data":"92c8994374331f891e754359b418f693315767a38f3f57de185d71dad79dbe79"}
May 28 07:59:29.321171 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.321055    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp" event={"ID":"520071c5-2f56-457c-9fc5-d833415d3248","Type":"ContainerStarted","Data":"e22bf696d2c5bdba3c5edd311c8ab80b912d6a69228ba99d02ebd524b59c72c2"}
May 28 07:59:29.322338 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.322316    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fjgcp" event={"ID":"7bce0101-b6a3-4da0-bab9-10edb7bcad0b","Type":"ContainerStarted","Data":"bd6d9c3e645bff974b2d3deb21c3d4567001ca73da1dd79b3e12fb6acb860aff"}
May 28 07:59:29.338744 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.338699    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-fjgcp" podStartSLOduration=2.52433758 podStartE2EDuration="21.338687648s" podCreationTimestamp="2026-05-28 07:59:08 +0000 UTC" firstStartedPulling="2026-05-28 07:59:09.452132188 +0000 UTC m=+1.892204462" lastFinishedPulling="2026-05-28 07:59:28.266482248 +0000 UTC m=+20.706554530" observedRunningTime="2026-05-28 07:59:29.338230358 +0000 UTC m=+21.778302659" watchObservedRunningTime="2026-05-28 07:59:29.338687648 +0000 UTC m=+21.778759937"
May 28 07:59:29.339110 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.339081    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-248.ec2.internal" podStartSLOduration=21.339074133 podStartE2EDuration="21.339074133s" podCreationTimestamp="2026-05-28 07:59:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 07:59:29.323552408 +0000 UTC m=+21.763624696" watchObservedRunningTime="2026-05-28 07:59:29.339074133 +0000 UTC m=+21.779146455"
May 28 07:59:29.367257 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.367208    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nfsn2" podStartSLOduration=2.624408494 podStartE2EDuration="21.367193175s" podCreationTimestamp="2026-05-28 07:59:08 +0000 UTC" firstStartedPulling="2026-05-28 07:59:09.468588701 +0000 UTC m=+1.908660977" lastFinishedPulling="2026-05-28 07:59:28.211373388 +0000 UTC m=+20.651445658" observedRunningTime="2026-05-28 07:59:29.351285783 +0000 UTC m=+21.791358074" watchObservedRunningTime="2026-05-28 07:59:29.367193175 +0000 UTC m=+21.807265467"
May 28 07:59:29.367461 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.367432    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lfc7s" podStartSLOduration=2.611709449 podStartE2EDuration="21.367424336s" podCreationTimestamp="2026-05-28 07:59:08 +0000 UTC" firstStartedPulling="2026-05-28 07:59:09.48062729 +0000 UTC m=+1.920699560" lastFinishedPulling="2026-05-28 07:59:28.236342168 +0000 UTC m=+20.676414447" observedRunningTime="2026-05-28 07:59:29.367037582 +0000 UTC m=+21.807109895" watchObservedRunningTime="2026-05-28 07:59:29.367424336 +0000 UTC m=+21.807496626"
May 28 07:59:29.419846 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.419722    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q9tbf" podStartSLOduration=2.6743678859999997 podStartE2EDuration="21.419706785s" podCreationTimestamp="2026-05-28 07:59:08 +0000 UTC" firstStartedPulling="2026-05-28 07:59:09.435839789 +0000 UTC m=+1.875912059" lastFinishedPulling="2026-05-28 07:59:28.181178676 +0000 UTC m=+20.621250958" observedRunningTime="2026-05-28 07:59:29.419651035 +0000 UTC m=+21.859723325" watchObservedRunningTime="2026-05-28 07:59:29.419706785 +0000 UTC m=+21.859779078"
May 28 07:59:29.440457 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.440414    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-8vcnc" podStartSLOduration=2.690209457 podStartE2EDuration="21.440396313s" podCreationTimestamp="2026-05-28 07:59:08 +0000 UTC" firstStartedPulling="2026-05-28 07:59:09.483712675 +0000 UTC m=+1.923784947" lastFinishedPulling="2026-05-28 07:59:28.233899522 +0000 UTC m=+20.673971803" observedRunningTime="2026-05-28 07:59:29.440321608 +0000 UTC m=+21.880393896" watchObservedRunningTime="2026-05-28 07:59:29.440396313 +0000 UTC m=+21.880468602"
May 28 07:59:29.605513 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:29.605480    2551 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 28 07:59:30.142557 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:30.142425    2551 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-28T07:59:29.605508778Z","UUID":"40880968-2b77-4266-87c0-8a95a2120d15","Handler":null,"Name":"","Endpoint":""}
May 28 07:59:30.145428 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:30.145287    2551 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 28 07:59:30.145428 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:30.145320    2551 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 28 07:59:30.190633 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:30.190593    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:30.190799 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:30.190604    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:30.190799 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:30.190735    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cnjvq" podUID="a659431e-376b-4589-8ae4-27694a9b763f"
May 28 07:59:30.190799 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:30.190762    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:30.326064 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:30.326030    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p6zng" event={"ID":"1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9","Type":"ContainerStarted","Data":"1061f9498d65b833f4d0b0eb80ca53042e0db00271f0461d49ef7339d57610bd"}
May 28 07:59:30.327792 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:30.327762    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal" event={"ID":"5062c11c04938e57cbece55012367eff","Type":"ContainerStarted","Data":"e09f7b624eff674502a999267a0ebe8997828b8531675988c27dc9c00eac082f"}
May 28 07:59:30.329405 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:30.329348    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj" event={"ID":"6fcecffb-0699-457c-a17f-be7801546dd8","Type":"ContainerStarted","Data":"f55ab22017f98189b14c55791df448018180c1e6592041eab592c699dd0d6157"}
May 28 07:59:30.339803 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:30.339761    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p6zng" podStartSLOduration=3.569034301 podStartE2EDuration="22.339746111s" podCreationTimestamp="2026-05-28 07:59:08 +0000 UTC" firstStartedPulling="2026-05-28 07:59:09.410463782 +0000 UTC m=+1.850536051" lastFinishedPulling="2026-05-28 07:59:28.181175584 +0000 UTC m=+20.621247861" observedRunningTime="2026-05-28 07:59:30.339559889 +0000 UTC m=+22.779632191" watchObservedRunningTime="2026-05-28 07:59:30.339746111 +0000 UTC m=+22.779818433"
May 28 07:59:30.353639 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:30.353595    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-248.ec2.internal" podStartSLOduration=22.353579659 podStartE2EDuration="22.353579659s" podCreationTimestamp="2026-05-28 07:59:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 07:59:30.353502776 +0000 UTC m=+22.793575066" watchObservedRunningTime="2026-05-28 07:59:30.353579659 +0000 UTC m=+22.793651949"
May 28 07:59:31.053051 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:31.053016    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nfsn2"
May 28 07:59:31.053855 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:31.053826    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nfsn2"
May 28 07:59:31.190374 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:31.190339    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:31.190556 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:31.190488    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:31.332964 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:31.332873    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj" event={"ID":"6fcecffb-0699-457c-a17f-be7801546dd8","Type":"ContainerStarted","Data":"e084bcc9149ca705cc7914ca98372615beb445bcda3c819f7ef43cb42cae775c"}
May 28 07:59:31.335569 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:31.335539    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp" event={"ID":"520071c5-2f56-457c-9fc5-d833415d3248","Type":"ContainerStarted","Data":"3d4a8fae3f71782b53e2f58d52230ae301444f0957d70538426fe98c32cd5541"}
May 28 07:59:31.335907 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:31.335873    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nfsn2"
May 28 07:59:31.336187 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:31.336164    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nfsn2"
May 28 07:59:31.348271 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:31.348235    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mq2lj" podStartSLOduration=2.44758034 podStartE2EDuration="23.348224493s" podCreationTimestamp="2026-05-28 07:59:08 +0000 UTC" firstStartedPulling="2026-05-28 07:59:09.472273806 +0000 UTC m=+1.912346076" lastFinishedPulling="2026-05-28 07:59:30.372917947 +0000 UTC m=+22.812990229" observedRunningTime="2026-05-28 07:59:31.348083403 +0000 UTC m=+23.788155691" watchObservedRunningTime="2026-05-28 07:59:31.348224493 +0000 UTC m=+23.788296782"
May 28 07:59:32.191063 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:32.191026    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:32.191253 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:32.191075    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:32.191253 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:32.191163    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:32.191378 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:32.191305    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cnjvq" podUID="a659431e-376b-4589-8ae4-27694a9b763f"
May 28 07:59:33.191079 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:33.191047    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:33.191534 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:33.191182    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:34.191057 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:34.190879    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:34.191223 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:34.190879    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:34.191223 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:34.191114    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:34.191223 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:34.191188    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cnjvq" podUID="a659431e-376b-4589-8ae4-27694a9b763f"
May 28 07:59:34.342513 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:34.342476    2551 generic.go:358] "Generic (PLEG): container finished" podID="64ffc18e-c32d-4d14-869f-fe837ec83037" containerID="43b48fd9b7a1498b5a5420bfb0c186bda65db5d0a73b17d17a7653c34f3b8795" exitCode=0
May 28 07:59:34.342670 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:34.342558    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcbtq" event={"ID":"64ffc18e-c32d-4d14-869f-fe837ec83037","Type":"ContainerDied","Data":"43b48fd9b7a1498b5a5420bfb0c186bda65db5d0a73b17d17a7653c34f3b8795"}
May 28 07:59:34.345637 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:34.345614    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp" event={"ID":"520071c5-2f56-457c-9fc5-d833415d3248","Type":"ContainerStarted","Data":"940cbb8fc5e254e3cb83b86f1a2371aab3d19e757a975b899d2108485d17f1ea"}
May 28 07:59:34.345924 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:34.345907    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:34.361222 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:34.361203    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:34.385688 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:34.385646    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp" podStartSLOduration=7.478653844 podStartE2EDuration="26.385633876s" podCreationTimestamp="2026-05-28 07:59:08 +0000 UTC" firstStartedPulling="2026-05-28 07:59:09.46116515 +0000 UTC m=+1.901237429" lastFinishedPulling="2026-05-28 07:59:28.368145194 +0000 UTC m=+20.808217461" observedRunningTime="2026-05-28 07:59:34.383881093 +0000 UTC m=+26.823953381" watchObservedRunningTime="2026-05-28 07:59:34.385633876 +0000 UTC m=+26.825706166"
May 28 07:59:35.190681 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:35.190523    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:35.190813 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:35.190761    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:35.348928 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:35.348821    2551 generic.go:358] "Generic (PLEG): container finished" podID="64ffc18e-c32d-4d14-869f-fe837ec83037" containerID="af446e2b8fb7bf31ab5301b2ee2b4240a2d02b274d010ef28a053fbcaed895bb" exitCode=0
May 28 07:59:35.348928 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:35.348909    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcbtq" event={"ID":"64ffc18e-c32d-4d14-869f-fe837ec83037","Type":"ContainerDied","Data":"af446e2b8fb7bf31ab5301b2ee2b4240a2d02b274d010ef28a053fbcaed895bb"}
May 28 07:59:35.349439 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:35.349416    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:35.349488 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:35.349443    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:35.364752 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:35.364733    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 07:59:35.455701 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:35.454702    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cnjvq"]
May 28 07:59:35.455701 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:35.454850    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:35.455701 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:35.454986    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cnjvq" podUID="a659431e-376b-4589-8ae4-27694a9b763f"
May 28 07:59:35.457317 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:35.457278    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dfswh"]
May 28 07:59:35.457467 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:35.457421    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:35.457561 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:35.457535    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:35.457914 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:35.457891    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t782k"]
May 28 07:59:35.458023 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:35.457996    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:35.458109 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:35.458088    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:36.353252 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:36.353173    2551 generic.go:358] "Generic (PLEG): container finished" podID="64ffc18e-c32d-4d14-869f-fe837ec83037" containerID="f893d8082e450cb2251a34ab3fa102b9447568afe231cf54afc896d296962475" exitCode=0
May 28 07:59:36.353611 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:36.353269    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcbtq" event={"ID":"64ffc18e-c32d-4d14-869f-fe837ec83037","Type":"ContainerDied","Data":"f893d8082e450cb2251a34ab3fa102b9447568afe231cf54afc896d296962475"}
May 28 07:59:36.701643 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:36.701616    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret\") pod \"global-pull-secret-syncer-cnjvq\" (UID: \"a659431e-376b-4589-8ae4-27694a9b763f\") " pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:36.701809 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:36.701727    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 28 07:59:36.701809 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:36.701789    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret podName:a659431e-376b-4589-8ae4-27694a9b763f nodeName:}" failed. No retries permitted until 2026-05-28 07:59:52.701770145 +0000 UTC m=+45.141842412 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret") pod "global-pull-secret-syncer-cnjvq" (UID: "a659431e-376b-4589-8ae4-27694a9b763f") : object "kube-system"/"original-pull-secret" not registered
May 28 07:59:37.191233 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:37.191204    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:37.191349 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:37.191204    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:37.191349 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:37.191208    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:37.191349 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:37.191331    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:37.191529 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:37.191456    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cnjvq" podUID="a659431e-376b-4589-8ae4-27694a9b763f"
May 28 07:59:37.191591 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:37.191569    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:39.190519 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:39.190488    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:39.190941 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:39.190538    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:39.190941 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:39.190488    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:39.190941 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:39.190631    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 07:59:39.190941 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:39.190713    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t782k" podUID="b33f26fc-296c-4693-b53a-a88ccd6f2236"
May 28 07:59:39.190941 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:39.190804    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cnjvq" podUID="a659431e-376b-4589-8ae4-27694a9b763f"
May 28 07:59:40.346502 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.346433    2551 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-248.ec2.internal" event="NodeReady"
May 28 07:59:40.346970 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.346595    2551 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 28 07:59:40.387351 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.387317    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q99nf"]
May 28 07:59:40.415571 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.415539    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cpkhd"]
May 28 07:59:40.415712 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.415643    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q99nf"
May 28 07:59:40.418309 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.418284    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 28 07:59:40.418456 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.418287    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 28 07:59:40.418456 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.418409    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4hh68\""
May 28 07:59:40.435946 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.435925    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q99nf"]
May 28 07:59:40.436063 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.435954    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cpkhd"]
May 28 07:59:40.436063 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.436010    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 07:59:40.438484 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.438464    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 28 07:59:40.438591 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.438509    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 28 07:59:40.438591 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.438530    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-47xg9\""
May 28 07:59:40.438959 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.438787    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 28 07:59:40.531944 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.531897    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7z29\" (UniqueName: \"kubernetes.io/projected/483c9981-b25f-41e0-92e1-d8ebcf6d6966-kube-api-access-x7z29\") pod \"ingress-canary-cpkhd\" (UID: \"483c9981-b25f-41e0-92e1-d8ebcf6d6966\") " pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 07:59:40.531944 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.531942    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa6c56b-06fe-483b-9755-25d6f4930f41-config-volume\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:40.532176 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.531979    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert\") pod \"ingress-canary-cpkhd\" (UID: \"483c9981-b25f-41e0-92e1-d8ebcf6d6966\") " pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 07:59:40.532176 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.531999    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:40.532176 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.532067    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa6c56b-06fe-483b-9755-25d6f4930f41-tmp-dir\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:40.532176 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.532102    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55s9d\" (UniqueName: \"kubernetes.io/projected/aaa6c56b-06fe-483b-9755-25d6f4930f41-kube-api-access-55s9d\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:40.632932 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.632842    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x7z29\" (UniqueName: \"kubernetes.io/projected/483c9981-b25f-41e0-92e1-d8ebcf6d6966-kube-api-access-x7z29\") pod \"ingress-canary-cpkhd\" (UID: \"483c9981-b25f-41e0-92e1-d8ebcf6d6966\") " pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 07:59:40.632932 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.632884    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa6c56b-06fe-483b-9755-25d6f4930f41-config-volume\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:40.632932 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.632925    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert\") pod \"ingress-canary-cpkhd\" (UID: \"483c9981-b25f-41e0-92e1-d8ebcf6d6966\") " pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 07:59:40.633220 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.632945    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:40.633220 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.632978    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa6c56b-06fe-483b-9755-25d6f4930f41-tmp-dir\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:40.633220 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.632999    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-55s9d\" (UniqueName: \"kubernetes.io/projected/aaa6c56b-06fe-483b-9755-25d6f4930f41-kube-api-access-55s9d\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:40.633220 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:40.633103    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 07:59:40.633220 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:40.633174    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls podName:aaa6c56b-06fe-483b-9755-25d6f4930f41 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:41.133151832 +0000 UTC m=+33.573224115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls") pod "dns-default-q99nf" (UID: "aaa6c56b-06fe-483b-9755-25d6f4930f41") : secret "dns-default-metrics-tls" not found
May 28 07:59:40.633469 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:40.633265    2551 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 07:59:40.633469 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:40.633305    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert podName:483c9981-b25f-41e0-92e1-d8ebcf6d6966 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:41.133295446 +0000 UTC m=+33.573367729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert") pod "ingress-canary-cpkhd" (UID: "483c9981-b25f-41e0-92e1-d8ebcf6d6966") : secret "canary-serving-cert" not found
May 28 07:59:40.633622 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.633602    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa6c56b-06fe-483b-9755-25d6f4930f41-tmp-dir\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:40.633826 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.633811    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa6c56b-06fe-483b-9755-25d6f4930f41-config-volume\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:40.646708 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.644339    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55s9d\" (UniqueName: \"kubernetes.io/projected/aaa6c56b-06fe-483b-9755-25d6f4930f41-kube-api-access-55s9d\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:40.646708 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.644472    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7z29\" (UniqueName: \"kubernetes.io/projected/483c9981-b25f-41e0-92e1-d8ebcf6d6966-kube-api-access-x7z29\") pod \"ingress-canary-cpkhd\" (UID: \"483c9981-b25f-41e0-92e1-d8ebcf6d6966\") " pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 07:59:40.835395 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.835355    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs\") pod \"network-metrics-daemon-dfswh\" (UID: \"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd\") " pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:40.835582 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:40.835530    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 07:59:40.835649 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:40.835611    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs podName:2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd nodeName:}" failed. No retries permitted until 2026-05-28 08:00:12.835590687 +0000 UTC m=+65.275662967 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs") pod "network-metrics-daemon-dfswh" (UID: "2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 28 07:59:40.936828 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:40.936746    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmwq\" (UniqueName: \"kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq\") pod \"network-check-target-t782k\" (UID: \"b33f26fc-296c-4693-b53a-a88ccd6f2236\") " pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:40.937035 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:40.936933    2551 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 28 07:59:40.937035 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:40.936960    2551 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 28 07:59:40.937035 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:40.936976    2551 projected.go:194] Error preparing data for projected volume kube-api-access-ggmwq for pod openshift-network-diagnostics/network-check-target-t782k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 07:59:40.937194 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:40.937047    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq podName:b33f26fc-296c-4693-b53a-a88ccd6f2236 nodeName:}" failed. No retries permitted until 2026-05-28 08:00:12.937027307 +0000 UTC m=+65.377099598 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ggmwq" (UniqueName: "kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq") pod "network-check-target-t782k" (UID: "b33f26fc-296c-4693-b53a-a88ccd6f2236") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 28 07:59:41.138257 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:41.138220    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert\") pod \"ingress-canary-cpkhd\" (UID: \"483c9981-b25f-41e0-92e1-d8ebcf6d6966\") " pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 07:59:41.138257 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:41.138257    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:41.138509 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:41.138372    2551 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 07:59:41.138509 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:41.138435    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 07:59:41.138509 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:41.138447    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert podName:483c9981-b25f-41e0-92e1-d8ebcf6d6966 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:42.138428081 +0000 UTC m=+34.578500353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert") pod "ingress-canary-cpkhd" (UID: "483c9981-b25f-41e0-92e1-d8ebcf6d6966") : secret "canary-serving-cert" not found
May 28 07:59:41.138509 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:41.138467    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls podName:aaa6c56b-06fe-483b-9755-25d6f4930f41 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:42.138457589 +0000 UTC m=+34.578529857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls") pod "dns-default-q99nf" (UID: "aaa6c56b-06fe-483b-9755-25d6f4930f41") : secret "dns-default-metrics-tls" not found
May 28 07:59:41.191060 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:41.190984    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 07:59:41.191220 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:41.191117    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 07:59:41.191220 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:41.191153    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:41.193787 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:41.193764    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 28 07:59:41.193911 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:41.193798    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 28 07:59:41.193911 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:41.193903    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 28 07:59:41.194074 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:41.194057    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fxd67\""
May 28 07:59:41.194149 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:41.194126    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q9jcs\""
May 28 07:59:41.194240 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:41.194220    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 28 07:59:42.146720 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:42.146684    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert\") pod \"ingress-canary-cpkhd\" (UID: \"483c9981-b25f-41e0-92e1-d8ebcf6d6966\") " pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 07:59:42.146720 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:42.146724    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:42.147328 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:42.146839    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 07:59:42.147328 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:42.146842    2551 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 07:59:42.147328 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:42.146921    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls podName:aaa6c56b-06fe-483b-9755-25d6f4930f41 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:44.146902668 +0000 UTC m=+36.586974937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls") pod "dns-default-q99nf" (UID: "aaa6c56b-06fe-483b-9755-25d6f4930f41") : secret "dns-default-metrics-tls" not found
May 28 07:59:42.147328 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:42.146940    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert podName:483c9981-b25f-41e0-92e1-d8ebcf6d6966 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:44.14693238 +0000 UTC m=+36.587004661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert") pod "ingress-canary-cpkhd" (UID: "483c9981-b25f-41e0-92e1-d8ebcf6d6966") : secret "canary-serving-cert" not found
May 28 07:59:43.368662 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:43.368632    2551 generic.go:358] "Generic (PLEG): container finished" podID="64ffc18e-c32d-4d14-869f-fe837ec83037" containerID="b22432d1b90006194cc2129a2535ff16a1b76473473bb80944455932ec24d565" exitCode=0
May 28 07:59:43.369172 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:43.368668    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcbtq" event={"ID":"64ffc18e-c32d-4d14-869f-fe837ec83037","Type":"ContainerDied","Data":"b22432d1b90006194cc2129a2535ff16a1b76473473bb80944455932ec24d565"}
May 28 07:59:44.163632 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:44.163602    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert\") pod \"ingress-canary-cpkhd\" (UID: \"483c9981-b25f-41e0-92e1-d8ebcf6d6966\") " pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 07:59:44.163632 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:44.163633    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:44.163822 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:44.163754    2551 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 07:59:44.163822 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:44.163795    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 07:59:44.163887 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:44.163832    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert podName:483c9981-b25f-41e0-92e1-d8ebcf6d6966 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:48.163811236 +0000 UTC m=+40.603883518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert") pod "ingress-canary-cpkhd" (UID: "483c9981-b25f-41e0-92e1-d8ebcf6d6966") : secret "canary-serving-cert" not found
May 28 07:59:44.163887 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:44.163847    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls podName:aaa6c56b-06fe-483b-9755-25d6f4930f41 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:48.163840659 +0000 UTC m=+40.603912926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls") pod "dns-default-q99nf" (UID: "aaa6c56b-06fe-483b-9755-25d6f4930f41") : secret "dns-default-metrics-tls" not found
May 28 07:59:44.373861 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:44.373809    2551 generic.go:358] "Generic (PLEG): container finished" podID="64ffc18e-c32d-4d14-869f-fe837ec83037" containerID="12f3c605c3bae9fdb039ecb9c97f2be8853d324111ebef3df53326354407776b" exitCode=0
May 28 07:59:44.374346 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:44.373864    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcbtq" event={"ID":"64ffc18e-c32d-4d14-869f-fe837ec83037","Type":"ContainerDied","Data":"12f3c605c3bae9fdb039ecb9c97f2be8853d324111ebef3df53326354407776b"}
May 28 07:59:45.378204 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:45.378125    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wcbtq" event={"ID":"64ffc18e-c32d-4d14-869f-fe837ec83037","Type":"ContainerStarted","Data":"b193e36777a5019b380687100739ad6a6f1adc21349101540ea7510dda7a82a2"}
May 28 07:59:45.400889 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:45.400845    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wcbtq" podStartSLOduration=4.166888401 podStartE2EDuration="37.400833742s" podCreationTimestamp="2026-05-28 07:59:08 +0000 UTC" firstStartedPulling="2026-05-28 07:59:09.390044857 +0000 UTC m=+1.830117127" lastFinishedPulling="2026-05-28 07:59:42.623990187 +0000 UTC m=+35.064062468" observedRunningTime="2026-05-28 07:59:45.39888083 +0000 UTC m=+37.838953120" watchObservedRunningTime="2026-05-28 07:59:45.400833742 +0000 UTC m=+37.840906030"
May 28 07:59:48.193407 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:48.193300    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert\") pod \"ingress-canary-cpkhd\" (UID: \"483c9981-b25f-41e0-92e1-d8ebcf6d6966\") " pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 07:59:48.193407 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:48.193354    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:48.193888 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:48.193541    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 07:59:48.193888 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:48.193608    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls podName:aaa6c56b-06fe-483b-9755-25d6f4930f41 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:56.19359061 +0000 UTC m=+48.633662892 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls") pod "dns-default-q99nf" (UID: "aaa6c56b-06fe-483b-9755-25d6f4930f41") : secret "dns-default-metrics-tls" not found
May 28 07:59:48.194057 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:48.194003    2551 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 07:59:48.194113 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:48.194072    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert podName:483c9981-b25f-41e0-92e1-d8ebcf6d6966 nodeName:}" failed. No retries permitted until 2026-05-28 07:59:56.194056066 +0000 UTC m=+48.634128342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert") pod "ingress-canary-cpkhd" (UID: "483c9981-b25f-41e0-92e1-d8ebcf6d6966") : secret "canary-serving-cert" not found
May 28 07:59:52.726363 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:52.726319    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret\") pod \"global-pull-secret-syncer-cnjvq\" (UID: \"a659431e-376b-4589-8ae4-27694a9b763f\") " pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:52.728953 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:52.728929    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a659431e-376b-4589-8ae4-27694a9b763f-original-pull-secret\") pod \"global-pull-secret-syncer-cnjvq\" (UID: \"a659431e-376b-4589-8ae4-27694a9b763f\") " pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:52.915733 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:52.915695    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cnjvq"
May 28 07:59:53.097160 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:53.097131    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cnjvq"]
May 28 07:59:53.101130 ip-10-0-136-248 kubenswrapper[2551]: W0528 07:59:53.101101    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda659431e_376b_4589_8ae4_27694a9b763f.slice/crio-5eac2511a579237bed3a338be9a695a815291fd898aacd7bb468a59b101e13c9 WatchSource:0}: Error finding container 5eac2511a579237bed3a338be9a695a815291fd898aacd7bb468a59b101e13c9: Status 404 returned error can't find the container with id 5eac2511a579237bed3a338be9a695a815291fd898aacd7bb468a59b101e13c9
May 28 07:59:53.392922 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:53.392848    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cnjvq" event={"ID":"a659431e-376b-4589-8ae4-27694a9b763f","Type":"ContainerStarted","Data":"5eac2511a579237bed3a338be9a695a815291fd898aacd7bb468a59b101e13c9"}
May 28 07:59:56.254141 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:56.254106    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert\") pod \"ingress-canary-cpkhd\" (UID: \"483c9981-b25f-41e0-92e1-d8ebcf6d6966\") " pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 07:59:56.254141 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:56.254143    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 07:59:56.254608 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:56.254237    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 07:59:56.254608 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:56.254260    2551 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 07:59:56.254608 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:56.254304    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls podName:aaa6c56b-06fe-483b-9755-25d6f4930f41 nodeName:}" failed. No retries permitted until 2026-05-28 08:00:12.254286724 +0000 UTC m=+64.694359006 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls") pod "dns-default-q99nf" (UID: "aaa6c56b-06fe-483b-9755-25d6f4930f41") : secret "dns-default-metrics-tls" not found
May 28 07:59:56.254608 ip-10-0-136-248 kubenswrapper[2551]: E0528 07:59:56.254319    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert podName:483c9981-b25f-41e0-92e1-d8ebcf6d6966 nodeName:}" failed. No retries permitted until 2026-05-28 08:00:12.254312877 +0000 UTC m=+64.694385143 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert") pod "ingress-canary-cpkhd" (UID: "483c9981-b25f-41e0-92e1-d8ebcf6d6966") : secret "canary-serving-cert" not found
May 28 07:59:58.404428 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:58.404394    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cnjvq" event={"ID":"a659431e-376b-4589-8ae4-27694a9b763f","Type":"ContainerStarted","Data":"b16b4a3f7c66d8d3ce045a56741d779bf5ba57efb445cfc2539a6b5135259544"}
May 28 07:59:58.419202 ip-10-0-136-248 kubenswrapper[2551]: I0528 07:59:58.419158    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cnjvq" podStartSLOduration=34.140791812 podStartE2EDuration="38.419144822s" podCreationTimestamp="2026-05-28 07:59:20 +0000 UTC" firstStartedPulling="2026-05-28 07:59:53.102811554 +0000 UTC m=+45.542883829" lastFinishedPulling="2026-05-28 07:59:57.381164567 +0000 UTC m=+49.821236839" observedRunningTime="2026-05-28 07:59:58.41873131 +0000 UTC m=+50.858803600" watchObservedRunningTime="2026-05-28 07:59:58.419144822 +0000 UTC m=+50.859217110"
May 28 08:00:07.365078 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:07.365048    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vvpbp"
May 28 08:00:12.265780 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:12.265739    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert\") pod \"ingress-canary-cpkhd\" (UID: \"483c9981-b25f-41e0-92e1-d8ebcf6d6966\") " pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 08:00:12.265780 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:12.265783    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 08:00:12.266212 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:00:12.265910    2551 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 08:00:12.266212 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:00:12.265925    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 08:00:12.266212 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:00:12.265978    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls podName:aaa6c56b-06fe-483b-9755-25d6f4930f41 nodeName:}" failed. No retries permitted until 2026-05-28 08:00:44.265961322 +0000 UTC m=+96.706033589 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls") pod "dns-default-q99nf" (UID: "aaa6c56b-06fe-483b-9755-25d6f4930f41") : secret "dns-default-metrics-tls" not found
May 28 08:00:12.266212 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:00:12.265991    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert podName:483c9981-b25f-41e0-92e1-d8ebcf6d6966 nodeName:}" failed. No retries permitted until 2026-05-28 08:00:44.265985576 +0000 UTC m=+96.706057843 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert") pod "ingress-canary-cpkhd" (UID: "483c9981-b25f-41e0-92e1-d8ebcf6d6966") : secret "canary-serving-cert" not found
May 28 08:00:12.870461 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:12.870422    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs\") pod \"network-metrics-daemon-dfswh\" (UID: \"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd\") " pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 08:00:12.873194 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:12.873173    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 28 08:00:12.881164 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:00:12.881137    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 28 08:00:12.881277 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:00:12.881202    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs podName:2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd nodeName:}" failed. No retries permitted until 2026-05-28 08:01:16.881185904 +0000 UTC m=+129.321258177 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs") pod "network-metrics-daemon-dfswh" (UID: "2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd") : secret "metrics-daemon-secret" not found
May 28 08:00:12.971606 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:12.971572    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmwq\" (UniqueName: \"kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq\") pod \"network-check-target-t782k\" (UID: \"b33f26fc-296c-4693-b53a-a88ccd6f2236\") " pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 08:00:12.974042 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:12.974022    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 28 08:00:12.983939 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:12.983917    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 28 08:00:12.994983 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:12.994959    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggmwq\" (UniqueName: \"kubernetes.io/projected/b33f26fc-296c-4693-b53a-a88ccd6f2236-kube-api-access-ggmwq\") pod \"network-check-target-t782k\" (UID: \"b33f26fc-296c-4693-b53a-a88ccd6f2236\") " pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 08:00:13.004871 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:13.004847    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fxd67\""
May 28 08:00:13.013801 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:13.013786    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 08:00:13.155123 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:13.155044    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t782k"]
May 28 08:00:13.158777 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:00:13.158748    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33f26fc_296c_4693_b53a_a88ccd6f2236.slice/crio-00ecc4fece29b3852db7daab3a1ddd6424c2df2b37cb024beeb9ca4e85e9e2e6 WatchSource:0}: Error finding container 00ecc4fece29b3852db7daab3a1ddd6424c2df2b37cb024beeb9ca4e85e9e2e6: Status 404 returned error can't find the container with id 00ecc4fece29b3852db7daab3a1ddd6424c2df2b37cb024beeb9ca4e85e9e2e6
May 28 08:00:13.433949 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:13.433864    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t782k" event={"ID":"b33f26fc-296c-4693-b53a-a88ccd6f2236","Type":"ContainerStarted","Data":"00ecc4fece29b3852db7daab3a1ddd6424c2df2b37cb024beeb9ca4e85e9e2e6"}
May 28 08:00:16.442564 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:16.442471    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t782k" event={"ID":"b33f26fc-296c-4693-b53a-a88ccd6f2236","Type":"ContainerStarted","Data":"b531f575260587b515b0dda965982d05d6e10c0d42a90197f123d5933503bafa"}
May 28 08:00:16.442968 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:16.442590    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 08:00:16.457105 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:16.457051    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t782k" podStartSLOduration=65.495958804 podStartE2EDuration="1m8.457037163s" podCreationTimestamp="2026-05-28 07:59:08 +0000 UTC" firstStartedPulling="2026-05-28 08:00:13.160530067 +0000 UTC m=+65.600602338" lastFinishedPulling="2026-05-28 08:00:16.121608427 +0000 UTC m=+68.561680697" observedRunningTime="2026-05-28 08:00:16.456763288 +0000 UTC m=+68.896835577" watchObservedRunningTime="2026-05-28 08:00:16.457037163 +0000 UTC m=+68.897109446"
May 28 08:00:44.288690 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:44.288640    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert\") pod \"ingress-canary-cpkhd\" (UID: \"483c9981-b25f-41e0-92e1-d8ebcf6d6966\") " pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 08:00:44.288690 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:44.288690    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 08:00:44.289331 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:00:44.288826    2551 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 08:00:44.289331 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:00:44.288851    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 08:00:44.289331 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:00:44.288905    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls podName:aaa6c56b-06fe-483b-9755-25d6f4930f41 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:48.28889017 +0000 UTC m=+160.728962436 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls") pod "dns-default-q99nf" (UID: "aaa6c56b-06fe-483b-9755-25d6f4930f41") : secret "dns-default-metrics-tls" not found
May 28 08:00:44.289331 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:00:44.288918    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert podName:483c9981-b25f-41e0-92e1-d8ebcf6d6966 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:48.288911696 +0000 UTC m=+160.728983962 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert") pod "ingress-canary-cpkhd" (UID: "483c9981-b25f-41e0-92e1-d8ebcf6d6966") : secret "canary-serving-cert" not found
May 28 08:00:47.447253 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:00:47.447220    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t782k"
May 28 08:01:16.913794 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:16.913729    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs\") pod \"network-metrics-daemon-dfswh\" (UID: \"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd\") " pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 08:01:16.914301 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:16.913863    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 28 08:01:16.914301 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:16.913946    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs podName:2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd nodeName:}" failed. No retries permitted until 2026-05-28 08:03:18.913930589 +0000 UTC m=+251.354002862 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs") pod "network-metrics-daemon-dfswh" (UID: "2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd") : secret "metrics-daemon-secret" not found
May 28 08:01:37.602439 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.602403    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"]
May 28 08:01:37.605182 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.605163    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:01:37.610005 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.609979    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 28 08:01:37.610105 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.610019    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
May 28 08:01:37.610164 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.609985    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 28 08:01:37.610164 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.610124    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bvbcn\""
May 28 08:01:37.610303 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.610230    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
May 28 08:01:37.612785 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.612766    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"]
May 28 08:01:37.709072 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.709031    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-6qh4c"]
May 28 08:01:37.712093 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.712075    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-t28bq"]
May 28 08:01:37.712237 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.712221    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-6qh4c"
May 28 08:01:37.714501 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.714470    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 28 08:01:37.714616 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.714522    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rprz4\""
May 28 08:01:37.714671 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.714663    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 28 08:01:37.714776 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.714762    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-56dd9859f6-76gdf"]
May 28 08:01:37.714937 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.714922    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:37.717106 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.717085    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 28 08:01:37.717399 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.717363    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 28 08:01:37.717539 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.717523    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:37.717605 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.717577    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 28 08:01:37.717727 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.717709    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-84h2g\""
May 28 08:01:37.717899 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.717713    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 28 08:01:37.720674 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.720653    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 28 08:01:37.720792 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.720770    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 28 08:01:37.720913 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.720738    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 28 08:01:37.721021 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.720999    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 28 08:01:37.721215 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.721199    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 28 08:01:37.721677 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.721659    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 28 08:01:37.721779 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.721705    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-swcjn\""
May 28 08:01:37.722210 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.722183    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-6qh4c"]
May 28 08:01:37.723754 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.723737    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 28 08:01:37.725187 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.725169    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-t28bq"]
May 28 08:01:37.738117 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.738093    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56dd9859f6-76gdf"]
May 28 08:01:37.752440 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.752415    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm247\" (UniqueName: \"kubernetes.io/projected/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-kube-api-access-bm247\") pod \"cluster-monitoring-operator-6c75959455-7q7sh\" (UID: \"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:01:37.752567 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.752447    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-7q7sh\" (UID: \"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:01:37.752567 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.752475    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7q7sh\" (UID: \"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:01:37.853161 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.853069    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:37.853161 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.853113    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6578639e-59ec-4e29-881d-f6d789b54b8f-trusted-ca\") pod \"console-operator-76dff48d59-t28bq\" (UID: \"6578639e-59ec-4e29-881d-f6d789b54b8f\") " pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:37.853161 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.853145    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j56ws\" (UniqueName: \"kubernetes.io/projected/a88b511e-eba6-4500-99f9-021664212882-kube-api-access-j56ws\") pod \"volume-data-source-validator-5cf8c89d64-6qh4c\" (UID: \"a88b511e-eba6-4500-99f9-021664212882\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-6qh4c"
May 28 08:01:37.853422 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.853169    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6578639e-59ec-4e29-881d-f6d789b54b8f-serving-cert\") pod \"console-operator-76dff48d59-t28bq\" (UID: \"6578639e-59ec-4e29-881d-f6d789b54b8f\") " pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:37.853422 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.853284    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bm247\" (UniqueName: \"kubernetes.io/projected/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-kube-api-access-bm247\") pod \"cluster-monitoring-operator-6c75959455-7q7sh\" (UID: \"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:01:37.853422 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.853317    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-default-certificate\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:37.853422 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.853337    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:37.853422 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.853362    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vv6z\" (UniqueName: \"kubernetes.io/projected/6578639e-59ec-4e29-881d-f6d789b54b8f-kube-api-access-5vv6z\") pod \"console-operator-76dff48d59-t28bq\" (UID: \"6578639e-59ec-4e29-881d-f6d789b54b8f\") " pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:37.853422 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.853400    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-7q7sh\" (UID: \"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:01:37.853691 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.853451    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7q7sh\" (UID: \"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:01:37.853691 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.853517    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-stats-auth\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:37.853691 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.853544    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtxtf\" (UniqueName: \"kubernetes.io/projected/0039c6a2-d3a2-40aa-9799-8237db852588-kube-api-access-jtxtf\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:37.853691 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.853572    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6578639e-59ec-4e29-881d-f6d789b54b8f-config\") pod \"console-operator-76dff48d59-t28bq\" (UID: \"6578639e-59ec-4e29-881d-f6d789b54b8f\") " pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:37.853691 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:37.853600    2551 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 08:01:37.853691 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:37.853662    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls podName:c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:38.353646754 +0000 UTC m=+150.793719020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-7q7sh" (UID: "c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4") : secret "cluster-monitoring-operator-tls" not found
May 28 08:01:37.854032 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.854014    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-7q7sh\" (UID: \"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:01:37.863407 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.863374    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm247\" (UniqueName: \"kubernetes.io/projected/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-kube-api-access-bm247\") pod \"cluster-monitoring-operator-6c75959455-7q7sh\" (UID: \"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:01:37.954798 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.954749    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:37.954798 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.954799    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6578639e-59ec-4e29-881d-f6d789b54b8f-trusted-ca\") pod \"console-operator-76dff48d59-t28bq\" (UID: \"6578639e-59ec-4e29-881d-f6d789b54b8f\") " pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:37.955050 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.954821    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j56ws\" (UniqueName: \"kubernetes.io/projected/a88b511e-eba6-4500-99f9-021664212882-kube-api-access-j56ws\") pod \"volume-data-source-validator-5cf8c89d64-6qh4c\" (UID: \"a88b511e-eba6-4500-99f9-021664212882\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-6qh4c"
May 28 08:01:37.955050 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.954919    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6578639e-59ec-4e29-881d-f6d789b54b8f-serving-cert\") pod \"console-operator-76dff48d59-t28bq\" (UID: \"6578639e-59ec-4e29-881d-f6d789b54b8f\") " pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:37.955050 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.955000    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-default-certificate\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:37.955050 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:37.955028    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle podName:0039c6a2-d3a2-40aa-9799-8237db852588 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:38.455003066 +0000 UTC m=+150.895075352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle") pod "router-default-56dd9859f6-76gdf" (UID: "0039c6a2-d3a2-40aa-9799-8237db852588") : configmap references non-existent config key: service-ca.crt
May 28 08:01:37.955259 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.955069    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:37.955259 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.955101    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5vv6z\" (UniqueName: \"kubernetes.io/projected/6578639e-59ec-4e29-881d-f6d789b54b8f-kube-api-access-5vv6z\") pod \"console-operator-76dff48d59-t28bq\" (UID: \"6578639e-59ec-4e29-881d-f6d789b54b8f\") " pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:37.955259 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:37.955159    2551 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 08:01:37.955259 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:37.955220    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs podName:0039c6a2-d3a2-40aa-9799-8237db852588 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:38.455202463 +0000 UTC m=+150.895274732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs") pod "router-default-56dd9859f6-76gdf" (UID: "0039c6a2-d3a2-40aa-9799-8237db852588") : secret "router-metrics-certs-default" not found
May 28 08:01:37.955484 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.955303    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-stats-auth\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:37.955484 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.955342    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jtxtf\" (UniqueName: \"kubernetes.io/projected/0039c6a2-d3a2-40aa-9799-8237db852588-kube-api-access-jtxtf\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:37.955484 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.955374    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6578639e-59ec-4e29-881d-f6d789b54b8f-config\") pod \"console-operator-76dff48d59-t28bq\" (UID: \"6578639e-59ec-4e29-881d-f6d789b54b8f\") " pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:37.955968 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.955944    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6578639e-59ec-4e29-881d-f6d789b54b8f-config\") pod \"console-operator-76dff48d59-t28bq\" (UID: \"6578639e-59ec-4e29-881d-f6d789b54b8f\") " pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:37.955968 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.955955    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6578639e-59ec-4e29-881d-f6d789b54b8f-trusted-ca\") pod \"console-operator-76dff48d59-t28bq\" (UID: \"6578639e-59ec-4e29-881d-f6d789b54b8f\") " pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:37.957422 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.957372    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6578639e-59ec-4e29-881d-f6d789b54b8f-serving-cert\") pod \"console-operator-76dff48d59-t28bq\" (UID: \"6578639e-59ec-4e29-881d-f6d789b54b8f\") " pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:37.957504 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.957444    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-default-certificate\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:37.957550 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.957510    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-stats-auth\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:37.965611 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.965590    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtxtf\" (UniqueName: \"kubernetes.io/projected/0039c6a2-d3a2-40aa-9799-8237db852588-kube-api-access-jtxtf\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:37.965611 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.965599    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vv6z\" (UniqueName: \"kubernetes.io/projected/6578639e-59ec-4e29-881d-f6d789b54b8f-kube-api-access-5vv6z\") pod \"console-operator-76dff48d59-t28bq\" (UID: \"6578639e-59ec-4e29-881d-f6d789b54b8f\") " pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:37.965760 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:37.965666    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j56ws\" (UniqueName: \"kubernetes.io/projected/a88b511e-eba6-4500-99f9-021664212882-kube-api-access-j56ws\") pod \"volume-data-source-validator-5cf8c89d64-6qh4c\" (UID: \"a88b511e-eba6-4500-99f9-021664212882\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-6qh4c"
May 28 08:01:38.023399 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:38.023336    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-6qh4c"
May 28 08:01:38.030244 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:38.030217    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:38.146486 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:38.146458    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-6qh4c"]
May 28 08:01:38.149038 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:01:38.149009    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88b511e_eba6_4500_99f9_021664212882.slice/crio-9c1cb95c73ab5ee1d8f562304a0e60a77b43a0a48576b7f4661838f40314efd1 WatchSource:0}: Error finding container 9c1cb95c73ab5ee1d8f562304a0e60a77b43a0a48576b7f4661838f40314efd1: Status 404 returned error can't find the container with id 9c1cb95c73ab5ee1d8f562304a0e60a77b43a0a48576b7f4661838f40314efd1
May 28 08:01:38.162998 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:38.162950    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-t28bq"]
May 28 08:01:38.166928 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:01:38.166903    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6578639e_59ec_4e29_881d_f6d789b54b8f.slice/crio-a1d2d81cef184012f0a8d8aa3faa339c2fd72b8deaf7b18a3f1da9f634c68f82 WatchSource:0}: Error finding container a1d2d81cef184012f0a8d8aa3faa339c2fd72b8deaf7b18a3f1da9f634c68f82: Status 404 returned error can't find the container with id a1d2d81cef184012f0a8d8aa3faa339c2fd72b8deaf7b18a3f1da9f634c68f82
May 28 08:01:38.364148 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:38.364055    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7q7sh\" (UID: \"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:01:38.364148 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:38.364103    2551 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 08:01:38.364362 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:38.364181    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls podName:c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:39.364160513 +0000 UTC m=+151.804232780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-7q7sh" (UID: "c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4") : secret "cluster-monitoring-operator-tls" not found
May 28 08:01:38.464676 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:38.464639    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:38.464838 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:38.464704    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:38.464838 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:38.464784    2551 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 08:01:38.464838 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:38.464822    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle podName:0039c6a2-d3a2-40aa-9799-8237db852588 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:39.464808979 +0000 UTC m=+151.904881246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle") pod "router-default-56dd9859f6-76gdf" (UID: "0039c6a2-d3a2-40aa-9799-8237db852588") : configmap references non-existent config key: service-ca.crt
May 28 08:01:38.464838 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:38.464838    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs podName:0039c6a2-d3a2-40aa-9799-8237db852588 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:39.464832152 +0000 UTC m=+151.904904419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs") pod "router-default-56dd9859f6-76gdf" (UID: "0039c6a2-d3a2-40aa-9799-8237db852588") : secret "router-metrics-certs-default" not found
May 28 08:01:38.596579 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:38.596504    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-t28bq" event={"ID":"6578639e-59ec-4e29-881d-f6d789b54b8f","Type":"ContainerStarted","Data":"a1d2d81cef184012f0a8d8aa3faa339c2fd72b8deaf7b18a3f1da9f634c68f82"}
May 28 08:01:38.597633 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:38.597599    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-6qh4c" event={"ID":"a88b511e-eba6-4500-99f9-021664212882","Type":"ContainerStarted","Data":"9c1cb95c73ab5ee1d8f562304a0e60a77b43a0a48576b7f4661838f40314efd1"}
May 28 08:01:39.372578 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:39.372527    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7q7sh\" (UID: \"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:01:39.373001 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:39.372691    2551 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 08:01:39.373001 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:39.372759    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls podName:c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:41.372743968 +0000 UTC m=+153.812816241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-7q7sh" (UID: "c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4") : secret "cluster-monitoring-operator-tls" not found
May 28 08:01:39.473491 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:39.473451    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:39.473672 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:39.473557    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:39.473672 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:39.473654    2551 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 08:01:39.473813 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:39.473732    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle podName:0039c6a2-d3a2-40aa-9799-8237db852588 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:41.473712714 +0000 UTC m=+153.913784994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle") pod "router-default-56dd9859f6-76gdf" (UID: "0039c6a2-d3a2-40aa-9799-8237db852588") : configmap references non-existent config key: service-ca.crt
May 28 08:01:39.473813 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:39.473756    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs podName:0039c6a2-d3a2-40aa-9799-8237db852588 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:41.47374627 +0000 UTC m=+153.913818586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs") pod "router-default-56dd9859f6-76gdf" (UID: "0039c6a2-d3a2-40aa-9799-8237db852588") : secret "router-metrics-certs-default" not found
May 28 08:01:40.602825 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:40.602799    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/0.log"
May 28 08:01:40.603221 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:40.602838    2551 generic.go:358] "Generic (PLEG): container finished" podID="6578639e-59ec-4e29-881d-f6d789b54b8f" containerID="b5643a5f4ca283672fa5e9a17f751782e8393e37d985fbc4e6509845465d9d98" exitCode=255
May 28 08:01:40.603221 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:40.602868    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-t28bq" event={"ID":"6578639e-59ec-4e29-881d-f6d789b54b8f","Type":"ContainerDied","Data":"b5643a5f4ca283672fa5e9a17f751782e8393e37d985fbc4e6509845465d9d98"}
May 28 08:01:40.603221 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:40.603123    2551 scope.go:117] "RemoveContainer" containerID="b5643a5f4ca283672fa5e9a17f751782e8393e37d985fbc4e6509845465d9d98"
May 28 08:01:40.604208 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:40.604180    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-6qh4c" event={"ID":"a88b511e-eba6-4500-99f9-021664212882","Type":"ContainerStarted","Data":"6ad7ded27aa5238313fd79ee6e6be372edd3c9d143746b3f578f1142e59fa23d"}
May 28 08:01:40.631463 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:40.631418    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-6qh4c" podStartSLOduration=1.764019523 podStartE2EDuration="3.63140516s" podCreationTimestamp="2026-05-28 08:01:37 +0000 UTC" firstStartedPulling="2026-05-28 08:01:38.150842906 +0000 UTC m=+150.590915173" lastFinishedPulling="2026-05-28 08:01:40.018228528 +0000 UTC m=+152.458300810" observedRunningTime="2026-05-28 08:01:40.630487818 +0000 UTC m=+153.070560108" watchObservedRunningTime="2026-05-28 08:01:40.63140516 +0000 UTC m=+153.071477446"
May 28 08:01:41.391791 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:41.391751    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7q7sh\" (UID: \"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:01:41.391967 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:41.391900    2551 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 08:01:41.392016 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:41.391980    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls podName:c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:45.391963483 +0000 UTC m=+157.832035755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-7q7sh" (UID: "c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4") : secret "cluster-monitoring-operator-tls" not found
May 28 08:01:41.492922 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:41.492887    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:41.493101 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:41.492956    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:41.493101 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:41.493068    2551 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 08:01:41.493101 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:41.493081    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle podName:0039c6a2-d3a2-40aa-9799-8237db852588 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:45.493058025 +0000 UTC m=+157.933130300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle") pod "router-default-56dd9859f6-76gdf" (UID: "0039c6a2-d3a2-40aa-9799-8237db852588") : configmap references non-existent config key: service-ca.crt
May 28 08:01:41.493275 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:41.493114    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs podName:0039c6a2-d3a2-40aa-9799-8237db852588 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:45.493102955 +0000 UTC m=+157.933175222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs") pod "router-default-56dd9859f6-76gdf" (UID: "0039c6a2-d3a2-40aa-9799-8237db852588") : secret "router-metrics-certs-default" not found
May 28 08:01:41.606823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:41.606796    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:01:41.607216 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:41.607094    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/0.log"
May 28 08:01:41.607216 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:41.607124    2551 generic.go:358] "Generic (PLEG): container finished" podID="6578639e-59ec-4e29-881d-f6d789b54b8f" containerID="643bfe97ffdb3eba9d4812a9e57dee22141e25c3f910b908e406e264a793e9ae" exitCode=255
May 28 08:01:41.607319 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:41.607229    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-t28bq" event={"ID":"6578639e-59ec-4e29-881d-f6d789b54b8f","Type":"ContainerDied","Data":"643bfe97ffdb3eba9d4812a9e57dee22141e25c3f910b908e406e264a793e9ae"}
May 28 08:01:41.607319 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:41.607277    2551 scope.go:117] "RemoveContainer" containerID="b5643a5f4ca283672fa5e9a17f751782e8393e37d985fbc4e6509845465d9d98"
May 28 08:01:41.607504 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:41.607477    2551 scope.go:117] "RemoveContainer" containerID="643bfe97ffdb3eba9d4812a9e57dee22141e25c3f910b908e406e264a793e9ae"
May 28 08:01:41.607676 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:41.607658    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-t28bq_openshift-console-operator(6578639e-59ec-4e29-881d-f6d789b54b8f)\"" pod="openshift-console-operator/console-operator-76dff48d59-t28bq" podUID="6578639e-59ec-4e29-881d-f6d789b54b8f"
May 28 08:01:42.610163 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:42.610137    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:01:42.610552 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:42.610492    2551 scope.go:117] "RemoveContainer" containerID="643bfe97ffdb3eba9d4812a9e57dee22141e25c3f910b908e406e264a793e9ae"
May 28 08:01:42.610667 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:42.610649    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-t28bq_openshift-console-operator(6578639e-59ec-4e29-881d-f6d789b54b8f)\"" pod="openshift-console-operator/console-operator-76dff48d59-t28bq" podUID="6578639e-59ec-4e29-881d-f6d789b54b8f"
May 28 08:01:43.426121 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:43.426068    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-q99nf" podUID="aaa6c56b-06fe-483b-9755-25d6f4930f41"
May 28 08:01:43.445822 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:43.445782    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-cpkhd" podUID="483c9981-b25f-41e0-92e1-d8ebcf6d6966"
May 28 08:01:43.612424 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:43.612393    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 08:01:43.612781 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:43.612400    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q99nf"
May 28 08:01:44.210011 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:44.209973    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-dfswh" podUID="2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd"
May 28 08:01:45.076176 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.076147    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-5xvxd"]
May 28 08:01:45.080250 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.080235    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-5xvxd"
May 28 08:01:45.082864 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.082840    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hvtq7\""
May 28 08:01:45.082864 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.082860    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 28 08:01:45.083876 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.083854    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 28 08:01:45.083998 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.083854    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 28 08:01:45.083998 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.083887    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 28 08:01:45.090940 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.090921    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-5xvxd"]
May 28 08:01:45.219332 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.219283    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr9qd\" (UniqueName: \"kubernetes.io/projected/29cb26de-9ec3-4170-a868-4b5b704f1e93-kube-api-access-sr9qd\") pod \"service-ca-67d667c7d-5xvxd\" (UID: \"29cb26de-9ec3-4170-a868-4b5b704f1e93\") " pod="openshift-service-ca/service-ca-67d667c7d-5xvxd"
May 28 08:01:45.219518 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.219355    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/29cb26de-9ec3-4170-a868-4b5b704f1e93-signing-key\") pod \"service-ca-67d667c7d-5xvxd\" (UID: \"29cb26de-9ec3-4170-a868-4b5b704f1e93\") " pod="openshift-service-ca/service-ca-67d667c7d-5xvxd"
May 28 08:01:45.219518 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.219477    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/29cb26de-9ec3-4170-a868-4b5b704f1e93-signing-cabundle\") pod \"service-ca-67d667c7d-5xvxd\" (UID: \"29cb26de-9ec3-4170-a868-4b5b704f1e93\") " pod="openshift-service-ca/service-ca-67d667c7d-5xvxd"
May 28 08:01:45.320836 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.320752    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/29cb26de-9ec3-4170-a868-4b5b704f1e93-signing-cabundle\") pod \"service-ca-67d667c7d-5xvxd\" (UID: \"29cb26de-9ec3-4170-a868-4b5b704f1e93\") " pod="openshift-service-ca/service-ca-67d667c7d-5xvxd"
May 28 08:01:45.320836 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.320806    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sr9qd\" (UniqueName: \"kubernetes.io/projected/29cb26de-9ec3-4170-a868-4b5b704f1e93-kube-api-access-sr9qd\") pod \"service-ca-67d667c7d-5xvxd\" (UID: \"29cb26de-9ec3-4170-a868-4b5b704f1e93\") " pod="openshift-service-ca/service-ca-67d667c7d-5xvxd"
May 28 08:01:45.321002 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.320863    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/29cb26de-9ec3-4170-a868-4b5b704f1e93-signing-key\") pod \"service-ca-67d667c7d-5xvxd\" (UID: \"29cb26de-9ec3-4170-a868-4b5b704f1e93\") " pod="openshift-service-ca/service-ca-67d667c7d-5xvxd"
May 28 08:01:45.321568 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.321545    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/29cb26de-9ec3-4170-a868-4b5b704f1e93-signing-cabundle\") pod \"service-ca-67d667c7d-5xvxd\" (UID: \"29cb26de-9ec3-4170-a868-4b5b704f1e93\") " pod="openshift-service-ca/service-ca-67d667c7d-5xvxd"
May 28 08:01:45.323245 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.323227    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/29cb26de-9ec3-4170-a868-4b5b704f1e93-signing-key\") pod \"service-ca-67d667c7d-5xvxd\" (UID: \"29cb26de-9ec3-4170-a868-4b5b704f1e93\") " pod="openshift-service-ca/service-ca-67d667c7d-5xvxd"
May 28 08:01:45.328158 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.328137    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr9qd\" (UniqueName: \"kubernetes.io/projected/29cb26de-9ec3-4170-a868-4b5b704f1e93-kube-api-access-sr9qd\") pod \"service-ca-67d667c7d-5xvxd\" (UID: \"29cb26de-9ec3-4170-a868-4b5b704f1e93\") " pod="openshift-service-ca/service-ca-67d667c7d-5xvxd"
May 28 08:01:45.388281 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.388254    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-5xvxd"
May 28 08:01:45.422226 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.422194    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7q7sh\" (UID: \"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:01:45.422362 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:45.422308    2551 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 08:01:45.422433 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:45.422364    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls podName:c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:53.422350672 +0000 UTC m=+165.862422939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-7q7sh" (UID: "c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4") : secret "cluster-monitoring-operator-tls" not found
May 28 08:01:45.507915 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.507883    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-5xvxd"]
May 28 08:01:45.512416 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:01:45.512368    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29cb26de_9ec3_4170_a868_4b5b704f1e93.slice/crio-110ed05e2d24b9837d1022409495d3d5978d1162837db54731de9c6844a7c050 WatchSource:0}: Error finding container 110ed05e2d24b9837d1022409495d3d5978d1162837db54731de9c6844a7c050: Status 404 returned error can't find the container with id 110ed05e2d24b9837d1022409495d3d5978d1162837db54731de9c6844a7c050
May 28 08:01:45.523057 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.523034    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:45.523147 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.523096    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:45.523235 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:45.523219    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle podName:0039c6a2-d3a2-40aa-9799-8237db852588 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:53.523201316 +0000 UTC m=+165.963273587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle") pod "router-default-56dd9859f6-76gdf" (UID: "0039c6a2-d3a2-40aa-9799-8237db852588") : configmap references non-existent config key: service-ca.crt
May 28 08:01:45.523275 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:45.523236    2551 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 08:01:45.523308 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:45.523292    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs podName:0039c6a2-d3a2-40aa-9799-8237db852588 nodeName:}" failed. No retries permitted until 2026-05-28 08:01:53.523279603 +0000 UTC m=+165.963351875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs") pod "router-default-56dd9859f6-76gdf" (UID: "0039c6a2-d3a2-40aa-9799-8237db852588") : secret "router-metrics-certs-default" not found
May 28 08:01:45.616970 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.616886    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-5xvxd" event={"ID":"29cb26de-9ec3-4170-a868-4b5b704f1e93","Type":"ContainerStarted","Data":"110ed05e2d24b9837d1022409495d3d5978d1162837db54731de9c6844a7c050"}
May 28 08:01:45.756707 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:45.756679    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8vcnc_91168ae0-b0ae-482d-be58-f879dcae55f1/dns-node-resolver/0.log"
May 28 08:01:46.954159 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:46.954129    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q9tbf_49ad95da-9ff8-462b-86aa-6f6eedf9ae3e/node-ca/0.log"
May 28 08:01:47.621605 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:47.621575    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-5xvxd" event={"ID":"29cb26de-9ec3-4170-a868-4b5b704f1e93","Type":"ContainerStarted","Data":"d89cce88ccdae3b78484433e3bd074d5b172c2b3ae13d869b9c0d06e971131e3"}
May 28 08:01:47.636039 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:47.635996    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-5xvxd" podStartSLOduration=0.744473962 podStartE2EDuration="2.635984246s" podCreationTimestamp="2026-05-28 08:01:45 +0000 UTC" firstStartedPulling="2026-05-28 08:01:45.514107929 +0000 UTC m=+157.954180199" lastFinishedPulling="2026-05-28 08:01:47.405618213 +0000 UTC m=+159.845690483" observedRunningTime="2026-05-28 08:01:47.634697757 +0000 UTC m=+160.074770046" watchObservedRunningTime="2026-05-28 08:01:47.635984246 +0000 UTC m=+160.076056534"
May 28 08:01:48.030953 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:48.030912    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:48.031326 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:48.030994    2551 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:01:48.031326 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:48.031279    2551 scope.go:117] "RemoveContainer" containerID="643bfe97ffdb3eba9d4812a9e57dee22141e25c3f910b908e406e264a793e9ae"
May 28 08:01:48.031479 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:48.031459    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-t28bq_openshift-console-operator(6578639e-59ec-4e29-881d-f6d789b54b8f)\"" pod="openshift-console-operator/console-operator-76dff48d59-t28bq" podUID="6578639e-59ec-4e29-881d-f6d789b54b8f"
May 28 08:01:48.344127 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:48.344040    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert\") pod \"ingress-canary-cpkhd\" (UID: \"483c9981-b25f-41e0-92e1-d8ebcf6d6966\") " pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 08:01:48.344127 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:48.344079    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 08:01:48.344329 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:48.344198    2551 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 28 08:01:48.344329 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:48.344256    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert podName:483c9981-b25f-41e0-92e1-d8ebcf6d6966 nodeName:}" failed. No retries permitted until 2026-05-28 08:03:50.34424216 +0000 UTC m=+282.784314431 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert") pod "ingress-canary-cpkhd" (UID: "483c9981-b25f-41e0-92e1-d8ebcf6d6966") : secret "canary-serving-cert" not found
May 28 08:01:48.344329 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:48.344278    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 28 08:01:48.344329 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:48.344328    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls podName:aaa6c56b-06fe-483b-9755-25d6f4930f41 nodeName:}" failed. No retries permitted until 2026-05-28 08:03:50.344315135 +0000 UTC m=+282.784387408 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls") pod "dns-default-q99nf" (UID: "aaa6c56b-06fe-483b-9755-25d6f4930f41") : secret "dns-default-metrics-tls" not found
May 28 08:01:48.624185 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:48.624110    2551 scope.go:117] "RemoveContainer" containerID="643bfe97ffdb3eba9d4812a9e57dee22141e25c3f910b908e406e264a793e9ae"
May 28 08:01:48.624374 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:48.624352    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-t28bq_openshift-console-operator(6578639e-59ec-4e29-881d-f6d789b54b8f)\"" pod="openshift-console-operator/console-operator-76dff48d59-t28bq" podUID="6578639e-59ec-4e29-881d-f6d789b54b8f"
May 28 08:01:53.482927 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:53.482835    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7q7sh\" (UID: \"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:01:53.483298 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:53.482953    2551 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 28 08:01:53.483298 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:53.483038    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls podName:c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4 nodeName:}" failed. No retries permitted until 2026-05-28 08:02:09.483021733 +0000 UTC m=+181.923094000 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-7q7sh" (UID: "c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4") : secret "cluster-monitoring-operator-tls" not found
May 28 08:01:53.583530 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:53.583489    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:53.583715 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:53.583593    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:01:53.583715 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:53.583650    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle podName:0039c6a2-d3a2-40aa-9799-8237db852588 nodeName:}" failed. No retries permitted until 2026-05-28 08:02:09.583630639 +0000 UTC m=+182.023702905 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle") pod "router-default-56dd9859f6-76gdf" (UID: "0039c6a2-d3a2-40aa-9799-8237db852588") : configmap references non-existent config key: service-ca.crt
May 28 08:01:53.583839 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:53.583720    2551 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 28 08:01:53.583839 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:01:53.583773    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs podName:0039c6a2-d3a2-40aa-9799-8237db852588 nodeName:}" failed. No retries permitted until 2026-05-28 08:02:09.583758649 +0000 UTC m=+182.023830922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs") pod "router-default-56dd9859f6-76gdf" (UID: "0039c6a2-d3a2-40aa-9799-8237db852588") : secret "router-metrics-certs-default" not found
May 28 08:01:56.191137 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:01:56.191103    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 08:02:03.191079 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.191051    2551 scope.go:117] "RemoveContainer" containerID="643bfe97ffdb3eba9d4812a9e57dee22141e25c3f910b908e406e264a793e9ae"
May 28 08:02:03.609821 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.609782    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xv4mt"]
May 28 08:02:03.613107 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.613087    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.615622 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.615595    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 28 08:02:03.616600 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.616572    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 28 08:02:03.616693 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.616674    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 28 08:02:03.616779 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.616679    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 28 08:02:03.616838 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.616787    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j77x8\""
May 28 08:02:03.628937 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.628914    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xv4mt"]
May 28 08:02:03.656947 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.656920    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:02:03.657083 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.656974    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-t28bq" event={"ID":"6578639e-59ec-4e29-881d-f6d789b54b8f","Type":"ContainerStarted","Data":"1ad5539940c0d69aedbe4c31f5c7a4fb504be7c982e0d547fb3543eeba3b07fa"}
May 28 08:02:03.657271 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.657251    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:02:03.662513 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.662476    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-t28bq"
May 28 08:02:03.664767 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.664740    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0291d14c-00af-44c4-b903-6d02d1daa8be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xv4mt\" (UID: \"0291d14c-00af-44c4-b903-6d02d1daa8be\") " pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.664894 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.664846    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0291d14c-00af-44c4-b903-6d02d1daa8be-crio-socket\") pod \"insights-runtime-extractor-xv4mt\" (UID: \"0291d14c-00af-44c4-b903-6d02d1daa8be\") " pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.664957 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.664934    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0291d14c-00af-44c4-b903-6d02d1daa8be-data-volume\") pod \"insights-runtime-extractor-xv4mt\" (UID: \"0291d14c-00af-44c4-b903-6d02d1daa8be\") " pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.665061 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.665044    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0291d14c-00af-44c4-b903-6d02d1daa8be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xv4mt\" (UID: \"0291d14c-00af-44c4-b903-6d02d1daa8be\") " pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.665132 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.665074    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zb9v\" (UniqueName: \"kubernetes.io/projected/0291d14c-00af-44c4-b903-6d02d1daa8be-kube-api-access-8zb9v\") pod \"insights-runtime-extractor-xv4mt\" (UID: \"0291d14c-00af-44c4-b903-6d02d1daa8be\") " pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.699967 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.699918    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-t28bq" podStartSLOduration=24.845461802 podStartE2EDuration="26.699899516s" podCreationTimestamp="2026-05-28 08:01:37 +0000 UTC" firstStartedPulling="2026-05-28 08:01:38.168428244 +0000 UTC m=+150.608500511" lastFinishedPulling="2026-05-28 08:01:40.022865953 +0000 UTC m=+152.462938225" observedRunningTime="2026-05-28 08:02:03.698672983 +0000 UTC m=+176.138745283" watchObservedRunningTime="2026-05-28 08:02:03.699899516 +0000 UTC m=+176.139971805"
May 28 08:02:03.755075 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.755047    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66575d684f-4nkxt"]
May 28 08:02:03.758009 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.757989    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.761161 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.761138    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 28 08:02:03.761287 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.761146    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 28 08:02:03.761523 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.761503    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 28 08:02:03.761612 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.761529    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mxgfh\""
May 28 08:02:03.765664 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.765638    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0291d14c-00af-44c4-b903-6d02d1daa8be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xv4mt\" (UID: \"0291d14c-00af-44c4-b903-6d02d1daa8be\") " pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.765773 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.765674    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8zb9v\" (UniqueName: \"kubernetes.io/projected/0291d14c-00af-44c4-b903-6d02d1daa8be-kube-api-access-8zb9v\") pod \"insights-runtime-extractor-xv4mt\" (UID: \"0291d14c-00af-44c4-b903-6d02d1daa8be\") " pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.765773 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.765751    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0291d14c-00af-44c4-b903-6d02d1daa8be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xv4mt\" (UID: \"0291d14c-00af-44c4-b903-6d02d1daa8be\") " pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.766116 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.765906    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0291d14c-00af-44c4-b903-6d02d1daa8be-crio-socket\") pod \"insights-runtime-extractor-xv4mt\" (UID: \"0291d14c-00af-44c4-b903-6d02d1daa8be\") " pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.766116 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.765987    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0291d14c-00af-44c4-b903-6d02d1daa8be-data-volume\") pod \"insights-runtime-extractor-xv4mt\" (UID: \"0291d14c-00af-44c4-b903-6d02d1daa8be\") " pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.766116 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.766041    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0291d14c-00af-44c4-b903-6d02d1daa8be-crio-socket\") pod \"insights-runtime-extractor-xv4mt\" (UID: \"0291d14c-00af-44c4-b903-6d02d1daa8be\") " pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.766327 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.766190    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0291d14c-00af-44c4-b903-6d02d1daa8be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xv4mt\" (UID: \"0291d14c-00af-44c4-b903-6d02d1daa8be\") " pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.766327 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.766303    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0291d14c-00af-44c4-b903-6d02d1daa8be-data-volume\") pod \"insights-runtime-extractor-xv4mt\" (UID: \"0291d14c-00af-44c4-b903-6d02d1daa8be\") " pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.768255 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.768235    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 28 08:02:03.768823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.768804    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0291d14c-00af-44c4-b903-6d02d1daa8be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xv4mt\" (UID: \"0291d14c-00af-44c4-b903-6d02d1daa8be\") " pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.772825 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.772802    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-597rs"]
May 28 08:02:03.776153 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.776134    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66575d684f-4nkxt"]
May 28 08:02:03.776254 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.776243    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-597rs"
May 28 08:02:03.778647 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.778624    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 28 08:02:03.778751 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.778676    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 28 08:02:03.778813 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.778789    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gtdfb\""
May 28 08:02:03.784705 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.784686    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zb9v\" (UniqueName: \"kubernetes.io/projected/0291d14c-00af-44c4-b903-6d02d1daa8be-kube-api-access-8zb9v\") pod \"insights-runtime-extractor-xv4mt\" (UID: \"0291d14c-00af-44c4-b903-6d02d1daa8be\") " pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.785071 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.785051    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-597rs"]
May 28 08:02:03.866616 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.866542    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-image-registry-private-configuration\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.866741 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.866632    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-bound-sa-token\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.866741 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.866683    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-ca-trust-extracted\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.866741 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.866701    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kp2d\" (UniqueName: \"kubernetes.io/projected/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-kube-api-access-9kp2d\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.866741 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.866727    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-installation-pull-secrets\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.866872 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.866765    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-trusted-ca\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.866872 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.866787    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nljpt\" (UniqueName: \"kubernetes.io/projected/8a78dae7-58a6-46a6-b6fa-c46e0df30aba-kube-api-access-nljpt\") pod \"downloads-57bd96b888-597rs\" (UID: \"8a78dae7-58a6-46a6-b6fa-c46e0df30aba\") " pod="openshift-console/downloads-57bd96b888-597rs"
May 28 08:02:03.866872 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.866856    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-registry-certificates\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.866964 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.866886    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-registry-tls\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.922270 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.922241    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xv4mt"
May 28 08:02:03.968803 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.967873    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-registry-certificates\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.968803 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.967924    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-registry-tls\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.968803 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.967969    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-image-registry-private-configuration\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.968803 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.968009    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-bound-sa-token\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.968803 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.968036    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-ca-trust-extracted\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.968803 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.968052    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9kp2d\" (UniqueName: \"kubernetes.io/projected/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-kube-api-access-9kp2d\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.968803 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.968079    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-installation-pull-secrets\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.968803 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.968130    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-trusted-ca\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.968803 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.968159    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nljpt\" (UniqueName: \"kubernetes.io/projected/8a78dae7-58a6-46a6-b6fa-c46e0df30aba-kube-api-access-nljpt\") pod \"downloads-57bd96b888-597rs\" (UID: \"8a78dae7-58a6-46a6-b6fa-c46e0df30aba\") " pod="openshift-console/downloads-57bd96b888-597rs"
May 28 08:02:03.969371 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.969342    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-registry-certificates\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.970136 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.969979    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-ca-trust-extracted\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.970339 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.970313    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-trusted-ca\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.972345 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.972257    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-image-registry-private-configuration\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.972472 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.972344    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-registry-tls\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.972636 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.972591    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-installation-pull-secrets\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.979123 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.979103    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nljpt\" (UniqueName: \"kubernetes.io/projected/8a78dae7-58a6-46a6-b6fa-c46e0df30aba-kube-api-access-nljpt\") pod \"downloads-57bd96b888-597rs\" (UID: \"8a78dae7-58a6-46a6-b6fa-c46e0df30aba\") " pod="openshift-console/downloads-57bd96b888-597rs"
May 28 08:02:03.979220 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.979149    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-bound-sa-token\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:03.979839 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:03.979819    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kp2d\" (UniqueName: \"kubernetes.io/projected/bc35ffa7-5b6f-45dd-93d5-5894c38b5259-kube-api-access-9kp2d\") pod \"image-registry-66575d684f-4nkxt\" (UID: \"bc35ffa7-5b6f-45dd-93d5-5894c38b5259\") " pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:04.043692 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:04.043659    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xv4mt"]
May 28 08:02:04.046478 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:02:04.046445    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0291d14c_00af_44c4_b903_6d02d1daa8be.slice/crio-981e1f08fd0b0e44e3c1bdebbb0474e4b905d91e7b98ddae13726d3b173daa14 WatchSource:0}: Error finding container 981e1f08fd0b0e44e3c1bdebbb0474e4b905d91e7b98ddae13726d3b173daa14: Status 404 returned error can't find the container with id 981e1f08fd0b0e44e3c1bdebbb0474e4b905d91e7b98ddae13726d3b173daa14
May 28 08:02:04.068418 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:04.068395    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:04.086139 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:04.086117    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-597rs"
May 28 08:02:04.201583 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:04.201553    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66575d684f-4nkxt"]
May 28 08:02:04.203873 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:02:04.203844    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc35ffa7_5b6f_45dd_93d5_5894c38b5259.slice/crio-0d157deb61f4592f50b4a19ff27b1232e0712015dbdb58441fb5fcc90a35a0ce WatchSource:0}: Error finding container 0d157deb61f4592f50b4a19ff27b1232e0712015dbdb58441fb5fcc90a35a0ce: Status 404 returned error can't find the container with id 0d157deb61f4592f50b4a19ff27b1232e0712015dbdb58441fb5fcc90a35a0ce
May 28 08:02:04.218053 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:04.218009    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-597rs"]
May 28 08:02:04.220926 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:02:04.220898    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a78dae7_58a6_46a6_b6fa_c46e0df30aba.slice/crio-6e89da0493664b24a1e2ecf2f8dd06a9bc6ab25d91c7acd8a0380f2a82150a60 WatchSource:0}: Error finding container 6e89da0493664b24a1e2ecf2f8dd06a9bc6ab25d91c7acd8a0380f2a82150a60: Status 404 returned error can't find the container with id 6e89da0493664b24a1e2ecf2f8dd06a9bc6ab25d91c7acd8a0380f2a82150a60
May 28 08:02:04.665038 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:04.664998    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xv4mt" event={"ID":"0291d14c-00af-44c4-b903-6d02d1daa8be","Type":"ContainerStarted","Data":"cd38dcd153bb6ccb4e19418a4ea02715ec1eeb9da63b0eeaaee159dc8ecbcc43"}
May 28 08:02:04.665206 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:04.665048    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xv4mt" event={"ID":"0291d14c-00af-44c4-b903-6d02d1daa8be","Type":"ContainerStarted","Data":"981e1f08fd0b0e44e3c1bdebbb0474e4b905d91e7b98ddae13726d3b173daa14"}
May 28 08:02:04.666417 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:04.666339    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66575d684f-4nkxt" event={"ID":"bc35ffa7-5b6f-45dd-93d5-5894c38b5259","Type":"ContainerStarted","Data":"8b6e378c644164494749b8e33a3c8516f130f2e4c7b2a953bd29d28f7c713b9a"}
May 28 08:02:04.666417 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:04.666394    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66575d684f-4nkxt" event={"ID":"bc35ffa7-5b6f-45dd-93d5-5894c38b5259","Type":"ContainerStarted","Data":"0d157deb61f4592f50b4a19ff27b1232e0712015dbdb58441fb5fcc90a35a0ce"}
May 28 08:02:04.666596 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:04.666487    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:04.667813 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:04.667785    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-597rs" event={"ID":"8a78dae7-58a6-46a6-b6fa-c46e0df30aba","Type":"ContainerStarted","Data":"6e89da0493664b24a1e2ecf2f8dd06a9bc6ab25d91c7acd8a0380f2a82150a60"}
May 28 08:02:04.686233 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:04.686193    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66575d684f-4nkxt" podStartSLOduration=1.6861797840000001 podStartE2EDuration="1.686179784s" podCreationTimestamp="2026-05-28 08:02:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:02:04.684615419 +0000 UTC m=+177.124687729" watchObservedRunningTime="2026-05-28 08:02:04.686179784 +0000 UTC m=+177.126252067"
May 28 08:02:05.673427 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:05.673359    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xv4mt" event={"ID":"0291d14c-00af-44c4-b903-6d02d1daa8be","Type":"ContainerStarted","Data":"6bdef47301ba5790d913f80e5763f6fac8d68156bde2e5b994dcd227f29585ce"}
May 28 08:02:06.677921 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:06.677879    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xv4mt" event={"ID":"0291d14c-00af-44c4-b903-6d02d1daa8be","Type":"ContainerStarted","Data":"474b6b05a3aba84c5c63735d53857e9030747d2c26305dbdbeffbd803444f240"}
May 28 08:02:06.693796 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:06.693740    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xv4mt" podStartSLOduration=1.685336708 podStartE2EDuration="3.693718619s" podCreationTimestamp="2026-05-28 08:02:03 +0000 UTC" firstStartedPulling="2026-05-28 08:02:04.110685877 +0000 UTC m=+176.550758148" lastFinishedPulling="2026-05-28 08:02:06.119067789 +0000 UTC m=+178.559140059" observedRunningTime="2026-05-28 08:02:06.692864809 +0000 UTC m=+179.132937099" watchObservedRunningTime="2026-05-28 08:02:06.693718619 +0000 UTC m=+179.133790908"
May 28 08:02:09.516358 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:09.516310    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7q7sh\" (UID: \"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:02:09.519161 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:09.519134    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7q7sh\" (UID: \"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:02:09.617420 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:09.617359    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:02:09.617599 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:09.617460    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:02:09.618153 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:09.618127    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0039c6a2-d3a2-40aa-9799-8237db852588-service-ca-bundle\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:02:09.619957 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:09.619932    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0039c6a2-d3a2-40aa-9799-8237db852588-metrics-certs\") pod \"router-default-56dd9859f6-76gdf\" (UID: \"0039c6a2-d3a2-40aa-9799-8237db852588\") " pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:02:09.716877 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:09.716844    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bvbcn\""
May 28 08:02:09.724989 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:09.724953    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"
May 28 08:02:09.838923 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:09.838893    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-swcjn\""
May 28 08:02:09.845953 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:09.845917    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:02:09.863727 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:09.863701    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh"]
May 28 08:02:09.866486 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:02:09.866458    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cf5bb2_5b95_4887_8fef_dbe7bdfd2ce4.slice/crio-a725a6225a6d0a01609182da472ab316e3abf142de8e800b2e1ef6133e0cda1a WatchSource:0}: Error finding container a725a6225a6d0a01609182da472ab316e3abf142de8e800b2e1ef6133e0cda1a: Status 404 returned error can't find the container with id a725a6225a6d0a01609182da472ab316e3abf142de8e800b2e1ef6133e0cda1a
May 28 08:02:09.997621 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:09.997590    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56dd9859f6-76gdf"]
May 28 08:02:10.000590 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:02:10.000557    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0039c6a2_d3a2_40aa_9799_8237db852588.slice/crio-de0c4a5c1d08c7f35c879ae7737afc23e43078337c274577e2a4aa07eece5873 WatchSource:0}: Error finding container de0c4a5c1d08c7f35c879ae7737afc23e43078337c274577e2a4aa07eece5873: Status 404 returned error can't find the container with id de0c4a5c1d08c7f35c879ae7737afc23e43078337c274577e2a4aa07eece5873
May 28 08:02:10.689483 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:10.689441    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56dd9859f6-76gdf" event={"ID":"0039c6a2-d3a2-40aa-9799-8237db852588","Type":"ContainerStarted","Data":"2581961d841e0a9f6e8dbb2a3db2546475ba7bfdbdafc539169d33888fbbb1ca"}
May 28 08:02:10.689483 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:10.689489    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56dd9859f6-76gdf" event={"ID":"0039c6a2-d3a2-40aa-9799-8237db852588","Type":"ContainerStarted","Data":"de0c4a5c1d08c7f35c879ae7737afc23e43078337c274577e2a4aa07eece5873"}
May 28 08:02:10.691303 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:10.691248    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh" event={"ID":"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4","Type":"ContainerStarted","Data":"a725a6225a6d0a01609182da472ab316e3abf142de8e800b2e1ef6133e0cda1a"}
May 28 08:02:10.710855 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:10.710768    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-56dd9859f6-76gdf" podStartSLOduration=33.710749794 podStartE2EDuration="33.710749794s" podCreationTimestamp="2026-05-28 08:01:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:02:10.708976035 +0000 UTC m=+183.149048327" watchObservedRunningTime="2026-05-28 08:02:10.710749794 +0000 UTC m=+183.150822083"
May 28 08:02:10.846262 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:10.846214    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:02:10.849301 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:10.849272    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:02:11.694531 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:11.694486    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:02:11.695942 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:11.695910    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-56dd9859f6-76gdf"
May 28 08:02:12.698434 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:12.698375    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh" event={"ID":"c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4","Type":"ContainerStarted","Data":"0f8239c916536d5cebe0c438c857f14adaa4071621b603b23b5c6cf3e782e64c"}
May 28 08:02:12.714766 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:12.714705    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7q7sh" podStartSLOduration=33.96191016 podStartE2EDuration="35.71468497s" podCreationTimestamp="2026-05-28 08:01:37 +0000 UTC" firstStartedPulling="2026-05-28 08:02:09.869326219 +0000 UTC m=+182.309398500" lastFinishedPulling="2026-05-28 08:02:11.622101027 +0000 UTC m=+184.062173310" observedRunningTime="2026-05-28 08:02:12.712921121 +0000 UTC m=+185.152993433" watchObservedRunningTime="2026-05-28 08:02:12.71468497 +0000 UTC m=+185.154757261"
May 28 08:02:20.721441 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:20.721369    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-597rs" event={"ID":"8a78dae7-58a6-46a6-b6fa-c46e0df30aba","Type":"ContainerStarted","Data":"5979325d5dafe5e75a7ac065061ae22211a294a3e775d2352ff72d080c415f27"}
May 28 08:02:20.721958 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:20.721612    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-597rs"
May 28 08:02:20.734246 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:20.734217    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-597rs"
May 28 08:02:20.739983 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:20.739937    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-597rs" podStartSLOduration=1.9118838679999999 podStartE2EDuration="17.739923464s" podCreationTimestamp="2026-05-28 08:02:03 +0000 UTC" firstStartedPulling="2026-05-28 08:02:04.222852459 +0000 UTC m=+176.662924744" lastFinishedPulling="2026-05-28 08:02:20.050892067 +0000 UTC m=+192.490964340" observedRunningTime="2026-05-28 08:02:20.737898083 +0000 UTC m=+193.177970373" watchObservedRunningTime="2026-05-28 08:02:20.739923464 +0000 UTC m=+193.179995753"
May 28 08:02:21.551346 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.551307    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"]
May 28 08:02:21.556151 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.556124    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"
May 28 08:02:21.560441 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.560418    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 28 08:02:21.560441 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.560437    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 28 08:02:21.560724 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.560706    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cgt8c\""
May 28 08:02:21.560977 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.560708    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 28 08:02:21.569708 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.569685    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"]
May 28 08:02:21.610228 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.610198    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fdhkw"]
May 28 08:02:21.614114 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.614087    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.618584 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.618560    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 28 08:02:21.619031 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.618774    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 28 08:02:21.619031 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.618624    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bks6g\""
May 28 08:02:21.619252 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.619238    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 28 08:02:21.726524 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.726454    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9251b57-4aa4-455d-9548-4aa1de30f61d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-dzv5g\" (UID: \"a9251b57-4aa4-455d-9548-4aa1de30f61d\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"
May 28 08:02:21.726524 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.726522    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9251b57-4aa4-455d-9548-4aa1de30f61d-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-dzv5g\" (UID: \"a9251b57-4aa4-455d-9548-4aa1de30f61d\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"
May 28 08:02:21.727026 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.726552    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30da5bc8-4838-4345-a058-65b26ae80ec8-metrics-client-ca\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.727026 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.726591    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-tls\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.727026 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.726661    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30da5bc8-4838-4345-a058-65b26ae80ec8-root\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.727026 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.726689    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.727026 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.726722    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30da5bc8-4838-4345-a058-65b26ae80ec8-sys\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.727026 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.726758    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j4vt\" (UniqueName: \"kubernetes.io/projected/a9251b57-4aa4-455d-9548-4aa1de30f61d-kube-api-access-8j4vt\") pod \"openshift-state-metrics-9c959559f-dzv5g\" (UID: \"a9251b57-4aa4-455d-9548-4aa1de30f61d\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"
May 28 08:02:21.727026 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.726785    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95wtm\" (UniqueName: \"kubernetes.io/projected/30da5bc8-4838-4345-a058-65b26ae80ec8-kube-api-access-95wtm\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.727026 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.726815    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-wtmp\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.727026 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.726864    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9251b57-4aa4-455d-9548-4aa1de30f61d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-dzv5g\" (UID: \"a9251b57-4aa4-455d-9548-4aa1de30f61d\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"
May 28 08:02:21.727026 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.726889    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-textfile\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.727026 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.726921    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-accelerators-collector-config\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.827622 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.827531    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30da5bc8-4838-4345-a058-65b26ae80ec8-sys\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.827953 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.827795    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30da5bc8-4838-4345-a058-65b26ae80ec8-sys\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.828143 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.828124    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8j4vt\" (UniqueName: \"kubernetes.io/projected/a9251b57-4aa4-455d-9548-4aa1de30f61d-kube-api-access-8j4vt\") pod \"openshift-state-metrics-9c959559f-dzv5g\" (UID: \"a9251b57-4aa4-455d-9548-4aa1de30f61d\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"
May 28 08:02:21.828750 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.828724    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-95wtm\" (UniqueName: \"kubernetes.io/projected/30da5bc8-4838-4345-a058-65b26ae80ec8-kube-api-access-95wtm\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.828924 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.828907    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-wtmp\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.829111 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.829077    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9251b57-4aa4-455d-9548-4aa1de30f61d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-dzv5g\" (UID: \"a9251b57-4aa4-455d-9548-4aa1de30f61d\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"
May 28 08:02:21.829261 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.829245    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-textfile\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.829416 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.829376    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-accelerators-collector-config\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.829552 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.829534    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9251b57-4aa4-455d-9548-4aa1de30f61d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-dzv5g\" (UID: \"a9251b57-4aa4-455d-9548-4aa1de30f61d\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"
May 28 08:02:21.829737 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.829704    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9251b57-4aa4-455d-9548-4aa1de30f61d-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-dzv5g\" (UID: \"a9251b57-4aa4-455d-9548-4aa1de30f61d\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"
May 28 08:02:21.829879 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.829864    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30da5bc8-4838-4345-a058-65b26ae80ec8-metrics-client-ca\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.833568 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.831815    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-tls\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.833568 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.831918    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30da5bc8-4838-4345-a058-65b26ae80ec8-root\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.833568 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.831947    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.833568 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.832193    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9251b57-4aa4-455d-9548-4aa1de30f61d-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-dzv5g\" (UID: \"a9251b57-4aa4-455d-9548-4aa1de30f61d\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"
May 28 08:02:21.833568 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.831461    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-accelerators-collector-config\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.833568 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.830640    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-wtmp\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.833568 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.832363    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30da5bc8-4838-4345-a058-65b26ae80ec8-root\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.833568 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:02:21.831547    2551 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
May 28 08:02:21.833568 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:02:21.832761    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9251b57-4aa4-455d-9548-4aa1de30f61d-openshift-state-metrics-tls podName:a9251b57-4aa4-455d-9548-4aa1de30f61d nodeName:}" failed. No retries permitted until 2026-05-28 08:02:22.332740793 +0000 UTC m=+194.772813074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a9251b57-4aa4-455d-9548-4aa1de30f61d-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-dzv5g" (UID: "a9251b57-4aa4-455d-9548-4aa1de30f61d") : secret "openshift-state-metrics-tls" not found
May 28 08:02:21.833568 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:02:21.832321    2551 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 28 08:02:21.833568 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:02:21.833014    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-tls podName:30da5bc8-4838-4345-a058-65b26ae80ec8 nodeName:}" failed. No retries permitted until 2026-05-28 08:02:22.332996106 +0000 UTC m=+194.773068374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-tls") pod "node-exporter-fdhkw" (UID: "30da5bc8-4838-4345-a058-65b26ae80ec8") : secret "node-exporter-tls" not found
May 28 08:02:21.833568 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.833519    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30da5bc8-4838-4345-a058-65b26ae80ec8-metrics-client-ca\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.842632 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.834814    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.842632 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.835049    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-textfile\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:21.847843 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.847822    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j4vt\" (UniqueName: \"kubernetes.io/projected/a9251b57-4aa4-455d-9548-4aa1de30f61d-kube-api-access-8j4vt\") pod \"openshift-state-metrics-9c959559f-dzv5g\" (UID: \"a9251b57-4aa4-455d-9548-4aa1de30f61d\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"
May 28 08:02:21.848594 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.848571    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9251b57-4aa4-455d-9548-4aa1de30f61d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-dzv5g\" (UID: \"a9251b57-4aa4-455d-9548-4aa1de30f61d\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"
May 28 08:02:21.853422 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:21.850118    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95wtm\" (UniqueName: \"kubernetes.io/projected/30da5bc8-4838-4345-a058-65b26ae80ec8-kube-api-access-95wtm\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:22.046765 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.046731    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-596d4f4dc7-jl225"]
May 28 08:02:22.051862 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.051830    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.056076 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.056045    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 28 08:02:22.056339 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.056318    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 28 08:02:22.056572 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.056554    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 28 08:02:22.057951 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.057931    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dfc4r\""
May 28 08:02:22.058239 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.058213    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 28 08:02:22.060607 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.060584    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 28 08:02:22.062895 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.062869    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-596d4f4dc7-jl225"]
May 28 08:02:22.068242 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.068222    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 28 08:02:22.134786 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.134711    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-serving-cert\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.134947 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.134789    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-trusted-ca-bundle\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.134947 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.134909    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-oauth-config\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.134947 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.134939    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-oauth-serving-cert\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.135101 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.135021    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-service-ca\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.135158 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.135117    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-config\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.135204 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.135186    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p826w\" (UniqueName: \"kubernetes.io/projected/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-kube-api-access-p826w\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.236966 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.236462    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-oauth-config\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.236966 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.236516    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-oauth-serving-cert\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.236966 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.236603    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-service-ca\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.236966 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.236647    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-config\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.236966 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.236692    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p826w\" (UniqueName: \"kubernetes.io/projected/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-kube-api-access-p826w\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.236966 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.236780    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-serving-cert\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.236966 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.236816    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-trusted-ca-bundle\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.238170 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.237739    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-config\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.238170 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.238113    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-service-ca\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.240435 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.239197    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-trusted-ca-bundle\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.240435 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.240081    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-oauth-serving-cert\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.240943 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.240920    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-oauth-config\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.243109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.243082    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-serving-cert\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.248293 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.248248    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p826w\" (UniqueName: \"kubernetes.io/projected/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-kube-api-access-p826w\") pod \"console-596d4f4dc7-jl225\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") " pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.338031 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.337984    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9251b57-4aa4-455d-9548-4aa1de30f61d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-dzv5g\" (UID: \"a9251b57-4aa4-455d-9548-4aa1de30f61d\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"
May 28 08:02:22.338237 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.338075    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-tls\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:22.340739 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.340710    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30da5bc8-4838-4345-a058-65b26ae80ec8-node-exporter-tls\") pod \"node-exporter-fdhkw\" (UID: \"30da5bc8-4838-4345-a058-65b26ae80ec8\") " pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:22.340900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.340743    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9251b57-4aa4-455d-9548-4aa1de30f61d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-dzv5g\" (UID: \"a9251b57-4aa4-455d-9548-4aa1de30f61d\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"
May 28 08:02:22.367968 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.367929    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:22.474895 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.474841    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"
May 28 08:02:22.528577 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.528455    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fdhkw"
May 28 08:02:22.529247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.529083    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-596d4f4dc7-jl225"]
May 28 08:02:22.650804 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.650741    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g"]
May 28 08:02:22.654959 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:02:22.654910    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9251b57_4aa4_455d_9548_4aa1de30f61d.slice/crio-d2e6d839de647df72fc889498deac0eb2a1b9d256ad0e90cf1584d8db6ad9200 WatchSource:0}: Error finding container d2e6d839de647df72fc889498deac0eb2a1b9d256ad0e90cf1584d8db6ad9200: Status 404 returned error can't find the container with id d2e6d839de647df72fc889498deac0eb2a1b9d256ad0e90cf1584d8db6ad9200
May 28 08:02:22.717544 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.717515    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 08:02:22.750176 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.750059    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fdhkw" event={"ID":"30da5bc8-4838-4345-a058-65b26ae80ec8","Type":"ContainerStarted","Data":"256b6f2ac3df7974331dfc772f02109f08be3b011565de17d97c26b9b8a397dc"}
May 28 08:02:22.750616 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.750595    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596d4f4dc7-jl225" event={"ID":"b50dcb32-5e7b-4087-a64d-7cc64aa1113c","Type":"ContainerStarted","Data":"d517104c42a12576ec666fab905e850ef346cef9ba023570e5c21858349fc7dc"}
May 28 08:02:22.750746 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.750733    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 08:02:22.750861 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.750845    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g" event={"ID":"a9251b57-4aa4-455d-9548-4aa1de30f61d","Type":"ContainerStarted","Data":"d2e6d839de647df72fc889498deac0eb2a1b9d256ad0e90cf1584d8db6ad9200"}
May 28 08:02:22.750969 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.750354    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.754730 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.753757    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 28 08:02:22.754730 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.753848    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 28 08:02:22.754730 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.754012    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 28 08:02:22.754730 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.754027    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 28 08:02:22.754730 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.753854    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 28 08:02:22.754730 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.754268    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 28 08:02:22.754730 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.754490    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 28 08:02:22.754730 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.754571    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 28 08:02:22.754730 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.754618    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xlk5w\""
May 28 08:02:22.759892 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.759689    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 28 08:02:22.844305 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.844204    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.844436 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.844319    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/471d349a-0799-4d4c-8e5d-e8a004868dd7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.844436 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.844353    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.844436 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.844404    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.844436 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.844432    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/471d349a-0799-4d4c-8e5d-e8a004868dd7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.844706 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.844482    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/471d349a-0799-4d4c-8e5d-e8a004868dd7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.844706 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.844535    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.844706 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.844568    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/471d349a-0799-4d4c-8e5d-e8a004868dd7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.844706 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.844601    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/471d349a-0799-4d4c-8e5d-e8a004868dd7-config-out\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.844706 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.844667    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.844706 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.844695    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-web-config\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.844986 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.844760    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlq9t\" (UniqueName: \"kubernetes.io/projected/471d349a-0799-4d4c-8e5d-e8a004868dd7-kube-api-access-tlq9t\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.844986 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.844787    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-config-volume\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.945698 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.945662    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.945854 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.945714    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-web-config\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.945854 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.945764    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tlq9t\" (UniqueName: \"kubernetes.io/projected/471d349a-0799-4d4c-8e5d-e8a004868dd7-kube-api-access-tlq9t\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.945854 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.945804    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-config-volume\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.946020 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.945852    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.946020 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.945878    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/471d349a-0799-4d4c-8e5d-e8a004868dd7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.946020 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.945906    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.946020 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.945936    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.946020 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.945965    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/471d349a-0799-4d4c-8e5d-e8a004868dd7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.946020 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.946009    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/471d349a-0799-4d4c-8e5d-e8a004868dd7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.946305 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.946056    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.946305 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.946091    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/471d349a-0799-4d4c-8e5d-e8a004868dd7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.946305 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.946125    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/471d349a-0799-4d4c-8e5d-e8a004868dd7-config-out\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.947347 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.946773    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/471d349a-0799-4d4c-8e5d-e8a004868dd7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.949245 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:02:22.948584    2551 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 28 08:02:22.949245 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:02:22.948658    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-main-tls podName:471d349a-0799-4d4c-8e5d-e8a004868dd7 nodeName:}" failed. No retries permitted until 2026-05-28 08:02:23.448640092 +0000 UTC m=+195.888712364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "471d349a-0799-4d4c-8e5d-e8a004868dd7") : secret "alertmanager-main-tls" not found
May 28 08:02:22.950217 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.949480    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/471d349a-0799-4d4c-8e5d-e8a004868dd7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.950217 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.950160    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/471d349a-0799-4d4c-8e5d-e8a004868dd7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.952272 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.952021    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.954678 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.953281    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.959521 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.955865    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/471d349a-0799-4d4c-8e5d-e8a004868dd7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.959521 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.956292    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-config-volume\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.959521 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.956732    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.959521 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.957673    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/471d349a-0799-4d4c-8e5d-e8a004868dd7-config-out\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.959521 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.957888    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.960052 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.960005    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-web-config\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:22.961759 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:22.961719    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlq9t\" (UniqueName: \"kubernetes.io/projected/471d349a-0799-4d4c-8e5d-e8a004868dd7-kube-api-access-tlq9t\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:23.451877 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.451831    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:23.452096 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:02:23.452076    2551 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 28 08:02:23.452174 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:02:23.452145    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-main-tls podName:471d349a-0799-4d4c-8e5d-e8a004868dd7 nodeName:}" failed. No retries permitted until 2026-05-28 08:02:24.452130119 +0000 UTC m=+196.892202406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "471d349a-0799-4d4c-8e5d-e8a004868dd7") : secret "alertmanager-main-tls" not found
May 28 08:02:23.561323 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.560652    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"]
May 28 08:02:23.578815 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.577640    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"]
May 28 08:02:23.578815 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.577832    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.582783 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.581079    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 28 08:02:23.582783 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.581346    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 28 08:02:23.582783 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.581567    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 28 08:02:23.582783 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.581773    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4d7bf\""
May 28 08:02:23.582783 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.582065    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 28 08:02:23.582783 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.582275    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6sime0qjtga83\""
May 28 08:02:23.583195 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.582850    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 28 08:02:23.736914 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.736792    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g" event={"ID":"a9251b57-4aa4-455d-9548-4aa1de30f61d","Type":"ContainerStarted","Data":"1492a08946f1c4b20931fe92121d0201b2c07967b936f3f07bdf68eb144b0647"}
May 28 08:02:23.736914 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.736837    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g" event={"ID":"a9251b57-4aa4-455d-9548-4aa1de30f61d","Type":"ContainerStarted","Data":"8d4b1949d8afae686e03dcea4bf7c92b2e2cc1b553edf9d5a6516e0ccf0cacd7"}
May 28 08:02:23.755093 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.754584    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.755093 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.754634    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.755093 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.754690    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.755093 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.754718    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhx45\" (UniqueName: \"kubernetes.io/projected/6f51145c-9285-4e23-9143-5a9e983d98b0-kube-api-access-nhx45\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.755093 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.754819    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.755093 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.754861    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f51145c-9285-4e23-9143-5a9e983d98b0-metrics-client-ca\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.755093 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.754885    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-grpc-tls\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.755093 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.754936    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-thanos-querier-tls\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.857274 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.856210    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.857274 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.856244    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.857274 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.856284    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.857274 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.856301    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nhx45\" (UniqueName: \"kubernetes.io/projected/6f51145c-9285-4e23-9143-5a9e983d98b0-kube-api-access-nhx45\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.857274 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.856362    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.857274 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.856413    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f51145c-9285-4e23-9143-5a9e983d98b0-metrics-client-ca\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.857274 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.856429    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-grpc-tls\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.857274 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.856460    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-thanos-querier-tls\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.858661 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.858582    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f51145c-9285-4e23-9143-5a9e983d98b0-metrics-client-ca\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.863809 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.863732    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-grpc-tls\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.865367 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.865313    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.867893 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.867834    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.868738 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.868697    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.871518 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.871476    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhx45\" (UniqueName: \"kubernetes.io/projected/6f51145c-9285-4e23-9143-5a9e983d98b0-kube-api-access-nhx45\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.884500 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.884477    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:23.900314 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:23.900286    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6f51145c-9285-4e23-9143-5a9e983d98b0-secret-thanos-querier-tls\") pod \"thanos-querier-6bb9978d9c-gn2xj\" (UID: \"6f51145c-9285-4e23-9143-5a9e983d98b0\") " pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:24.193527 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:24.193370    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:24.464548 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:24.463977    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:24.477404 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:24.477011    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:24.565854 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:24.565373    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 08:02:25.241327 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:25.241272    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 08:02:25.266986 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:25.266827    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"]
May 28 08:02:25.678418 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:25.678327    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66575d684f-4nkxt"
May 28 08:02:25.745217 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:25.744954    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fdhkw" event={"ID":"30da5bc8-4838-4345-a058-65b26ae80ec8","Type":"ContainerStarted","Data":"fd26b6d01bb408b067cb8e2454d5915b3a693cd1c19c65f47fe456a86d91b8f6"}
May 28 08:02:26.540753 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:02:26.540684    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471d349a_0799_4d4c_8e5d_e8a004868dd7.slice/crio-c4943f7fefd5c6e58ebdad64104948b3948c6fa97fd5a50afcfde3e9d90817fb WatchSource:0}: Error finding container c4943f7fefd5c6e58ebdad64104948b3948c6fa97fd5a50afcfde3e9d90817fb: Status 404 returned error can't find the container with id c4943f7fefd5c6e58ebdad64104948b3948c6fa97fd5a50afcfde3e9d90817fb
May 28 08:02:26.544295 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:02:26.544146    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f51145c_9285_4e23_9143_5a9e983d98b0.slice/crio-c7970b8e1cea2da658175c58784280ff619e8e06390bd288ca5388c1115c6adc WatchSource:0}: Error finding container c7970b8e1cea2da658175c58784280ff619e8e06390bd288ca5388c1115c6adc: Status 404 returned error can't find the container with id c7970b8e1cea2da658175c58784280ff619e8e06390bd288ca5388c1115c6adc
May 28 08:02:26.749869 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.749826    2551 generic.go:358] "Generic (PLEG): container finished" podID="30da5bc8-4838-4345-a058-65b26ae80ec8" containerID="fd26b6d01bb408b067cb8e2454d5915b3a693cd1c19c65f47fe456a86d91b8f6" exitCode=0
May 28 08:02:26.750077 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.749913    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fdhkw" event={"ID":"30da5bc8-4838-4345-a058-65b26ae80ec8","Type":"ContainerDied","Data":"fd26b6d01bb408b067cb8e2454d5915b3a693cd1c19c65f47fe456a86d91b8f6"}
May 28 08:02:26.751426 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.751375    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"471d349a-0799-4d4c-8e5d-e8a004868dd7","Type":"ContainerStarted","Data":"c4943f7fefd5c6e58ebdad64104948b3948c6fa97fd5a50afcfde3e9d90817fb"}
May 28 08:02:26.753896 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.753872    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g" event={"ID":"a9251b57-4aa4-455d-9548-4aa1de30f61d","Type":"ContainerStarted","Data":"79c279509a84b31a84b351985afdf46afe452941875d766304fd910f95b6e207"}
May 28 08:02:26.755004 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.754979    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj" event={"ID":"6f51145c-9285-4e23-9143-5a9e983d98b0","Type":"ContainerStarted","Data":"c7970b8e1cea2da658175c58784280ff619e8e06390bd288ca5388c1115c6adc"}
May 28 08:02:26.783278 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.783249    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-cc545fc4-mhqk4"]
May 28 08:02:26.796432 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.796347    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.800590 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.798628    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-dzv5g" podStartSLOduration=3.699072813 podStartE2EDuration="5.798612564s" podCreationTimestamp="2026-05-28 08:02:21 +0000 UTC" firstStartedPulling="2026-05-28 08:02:22.952112564 +0000 UTC m=+195.392184836" lastFinishedPulling="2026-05-28 08:02:25.051652305 +0000 UTC m=+197.491724587" observedRunningTime="2026-05-28 08:02:26.797682378 +0000 UTC m=+199.237754668" watchObservedRunningTime="2026-05-28 08:02:26.798612564 +0000 UTC m=+199.238684854"
May 28 08:02:26.800590 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.800515    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 28 08:02:26.800784 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.800598    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qtl2f\""
May 28 08:02:26.800784 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.800662    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 28 08:02:26.800784 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.800668    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 28 08:02:26.801003 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.800921    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 28 08:02:26.801093 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.801022    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 28 08:02:26.802806 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.802779    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-cc545fc4-mhqk4"]
May 28 08:02:26.805971 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.805943    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 28 08:02:26.890267 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.890231    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e2c6ff-5f87-406b-ae23-e8fbace620f4-serving-certs-ca-bundle\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.890491 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.890278    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/17e2c6ff-5f87-406b-ae23-e8fbace620f4-secret-telemeter-client\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.890491 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.890354    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17e2c6ff-5f87-406b-ae23-e8fbace620f4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.890491 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.890464    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx7bv\" (UniqueName: \"kubernetes.io/projected/17e2c6ff-5f87-406b-ae23-e8fbace620f4-kube-api-access-zx7bv\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.890653 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.890515    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17e2c6ff-5f87-406b-ae23-e8fbace620f4-metrics-client-ca\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.890653 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.890619    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/17e2c6ff-5f87-406b-ae23-e8fbace620f4-federate-client-tls\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.891538 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.891248    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e2c6ff-5f87-406b-ae23-e8fbace620f4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.891647 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.891574    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/17e2c6ff-5f87-406b-ae23-e8fbace620f4-telemeter-client-tls\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.992536 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.992509    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e2c6ff-5f87-406b-ae23-e8fbace620f4-serving-certs-ca-bundle\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.992653 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.992555    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/17e2c6ff-5f87-406b-ae23-e8fbace620f4-secret-telemeter-client\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.992653 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.992606    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17e2c6ff-5f87-406b-ae23-e8fbace620f4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.992653 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.992645    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zx7bv\" (UniqueName: \"kubernetes.io/projected/17e2c6ff-5f87-406b-ae23-e8fbace620f4-kube-api-access-zx7bv\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.992838 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.992673    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17e2c6ff-5f87-406b-ae23-e8fbace620f4-metrics-client-ca\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.992838 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.992724    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/17e2c6ff-5f87-406b-ae23-e8fbace620f4-federate-client-tls\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.992838 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.992806    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e2c6ff-5f87-406b-ae23-e8fbace620f4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.992975 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.992840    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/17e2c6ff-5f87-406b-ae23-e8fbace620f4-telemeter-client-tls\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.994344 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.993766    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17e2c6ff-5f87-406b-ae23-e8fbace620f4-metrics-client-ca\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.994920 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.994894    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e2c6ff-5f87-406b-ae23-e8fbace620f4-serving-certs-ca-bundle\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.995118 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.995095    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17e2c6ff-5f87-406b-ae23-e8fbace620f4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.996759 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.996714    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17e2c6ff-5f87-406b-ae23-e8fbace620f4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.997427 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.997071    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/17e2c6ff-5f87-406b-ae23-e8fbace620f4-telemeter-client-tls\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.997427 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.997350    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/17e2c6ff-5f87-406b-ae23-e8fbace620f4-secret-telemeter-client\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:26.997691 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:26.997648    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/17e2c6ff-5f87-406b-ae23-e8fbace620f4-federate-client-tls\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:27.001559 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:27.001537    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx7bv\" (UniqueName: \"kubernetes.io/projected/17e2c6ff-5f87-406b-ae23-e8fbace620f4-kube-api-access-zx7bv\") pod \"telemeter-client-cc545fc4-mhqk4\" (UID: \"17e2c6ff-5f87-406b-ae23-e8fbace620f4\") " pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:27.113396 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:27.113348    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4"
May 28 08:02:27.283673 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:27.283420    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-cc545fc4-mhqk4"]
May 28 08:02:27.292739 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:02:27.291740    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e2c6ff_5f87_406b_ae23_e8fbace620f4.slice/crio-aa08eaed3a4023dd60bbb0c619fd52902c0206a74b6cac660a3245c7eaebd175 WatchSource:0}: Error finding container aa08eaed3a4023dd60bbb0c619fd52902c0206a74b6cac660a3245c7eaebd175: Status 404 returned error can't find the container with id aa08eaed3a4023dd60bbb0c619fd52902c0206a74b6cac660a3245c7eaebd175
May 28 08:02:27.760469 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:27.760416    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4" event={"ID":"17e2c6ff-5f87-406b-ae23-e8fbace620f4","Type":"ContainerStarted","Data":"aa08eaed3a4023dd60bbb0c619fd52902c0206a74b6cac660a3245c7eaebd175"}
May 28 08:02:27.763862 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:27.763606    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fdhkw" event={"ID":"30da5bc8-4838-4345-a058-65b26ae80ec8","Type":"ContainerStarted","Data":"40de3e47c6590af93e6531193a419e8a20c6bd828c5385104208e8197ada09a9"}
May 28 08:02:27.763862 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:27.763638    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fdhkw" event={"ID":"30da5bc8-4838-4345-a058-65b26ae80ec8","Type":"ContainerStarted","Data":"213997ab12b144058a39c7bd203355c4f8e708eef55f38ae81030b767ac84285"}
May 28 08:02:27.767267 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:27.767207    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596d4f4dc7-jl225" event={"ID":"b50dcb32-5e7b-4087-a64d-7cc64aa1113c","Type":"ContainerStarted","Data":"2f44c186bb9b933856c709fd3c8d29a02e1d5414917f83a612969142d5a1bc0a"}
May 28 08:02:27.786187 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:27.786112    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fdhkw" podStartSLOduration=4.28611126 podStartE2EDuration="6.786093538s" podCreationTimestamp="2026-05-28 08:02:21 +0000 UTC" firstStartedPulling="2026-05-28 08:02:22.54862323 +0000 UTC m=+194.988695501" lastFinishedPulling="2026-05-28 08:02:25.04860551 +0000 UTC m=+197.488677779" observedRunningTime="2026-05-28 08:02:27.785855226 +0000 UTC m=+200.225927516" watchObservedRunningTime="2026-05-28 08:02:27.786093538 +0000 UTC m=+200.226165828"
May 28 08:02:27.808352 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:27.808285    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-596d4f4dc7-jl225" podStartSLOduration=1.428914523 podStartE2EDuration="5.808269855s" podCreationTimestamp="2026-05-28 08:02:22 +0000 UTC" firstStartedPulling="2026-05-28 08:02:22.543341647 +0000 UTC m=+194.983413920" lastFinishedPulling="2026-05-28 08:02:26.92269697 +0000 UTC m=+199.362769252" observedRunningTime="2026-05-28 08:02:27.807329582 +0000 UTC m=+200.247402090" watchObservedRunningTime="2026-05-28 08:02:27.808269855 +0000 UTC m=+200.248342146"
May 28 08:02:28.461575 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:28.461537    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-596d4f4dc7-jl225"]
May 28 08:02:29.774435 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:29.774327    2551 generic.go:358] "Generic (PLEG): container finished" podID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerID="259f2b12a51d5039249a168bcc614de7c89b0487dce91f3554423e82eb3ab5cb" exitCode=0
May 28 08:02:29.774865 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:29.774430    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"471d349a-0799-4d4c-8e5d-e8a004868dd7","Type":"ContainerDied","Data":"259f2b12a51d5039249a168bcc614de7c89b0487dce91f3554423e82eb3ab5cb"}
May 28 08:02:30.779658 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:30.779628    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj" event={"ID":"6f51145c-9285-4e23-9143-5a9e983d98b0","Type":"ContainerStarted","Data":"e99a107592a72abbd37b52a6a6e76a07b9bb67d9dc072ad5124449e5c4d533f0"}
May 28 08:02:30.780055 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:30.779664    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj" event={"ID":"6f51145c-9285-4e23-9143-5a9e983d98b0","Type":"ContainerStarted","Data":"35626b86e5d314ea64194e038d232f51063bddb9d0fc700acbf6b0fa91bf8798"}
May 28 08:02:30.780055 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:30.779673    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj" event={"ID":"6f51145c-9285-4e23-9143-5a9e983d98b0","Type":"ContainerStarted","Data":"afd18c4c9a9f12815c4d29c0d043e662ae76eefd999449a56cde6bd9362db467"}
May 28 08:02:30.781805 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:30.781763    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4" event={"ID":"17e2c6ff-5f87-406b-ae23-e8fbace620f4","Type":"ContainerStarted","Data":"90518cd27e62a4f2964a1d1504526b367738219d9acc45eec33909cd265ad94e"}
May 28 08:02:30.781805 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:30.781805    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4" event={"ID":"17e2c6ff-5f87-406b-ae23-e8fbace620f4","Type":"ContainerStarted","Data":"b2f068f3ab6e53ec4a14a746c3f0110debdbd77ae5e95163ef46b58c7090f03a"}
May 28 08:02:30.782050 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:30.781820    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4" event={"ID":"17e2c6ff-5f87-406b-ae23-e8fbace620f4","Type":"ContainerStarted","Data":"22bea330292861abab71be4c86120c9065bfe97dec7c0b7753eb14eb62cff31c"}
May 28 08:02:30.802697 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:30.802651    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-cc545fc4-mhqk4" podStartSLOduration=1.711220931 podStartE2EDuration="4.80263752s" podCreationTimestamp="2026-05-28 08:02:26 +0000 UTC" firstStartedPulling="2026-05-28 08:02:27.293936879 +0000 UTC m=+199.734009171" lastFinishedPulling="2026-05-28 08:02:30.385353487 +0000 UTC m=+202.825425760" observedRunningTime="2026-05-28 08:02:30.800072733 +0000 UTC m=+203.240145023" watchObservedRunningTime="2026-05-28 08:02:30.80263752 +0000 UTC m=+203.242709808"
May 28 08:02:32.368961 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:32.368936    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:32.793237 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:32.793190    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj" event={"ID":"6f51145c-9285-4e23-9143-5a9e983d98b0","Type":"ContainerStarted","Data":"abf328b5639241808874b0541841dfa381f44d36d996dfae84354452690b28e1"}
May 28 08:02:32.793237 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:32.793238    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj" event={"ID":"6f51145c-9285-4e23-9143-5a9e983d98b0","Type":"ContainerStarted","Data":"a2feb2b631b6b197df3225641d0a142c6de6a338330427c88b6c67d9832d47fd"}
May 28 08:02:32.793475 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:32.793251    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj" event={"ID":"6f51145c-9285-4e23-9143-5a9e983d98b0","Type":"ContainerStarted","Data":"81ad30285835fcaa56e6b2a650564b20c72663ae2d8724a7d685cf3307b737ad"}
May 28 08:02:32.793475 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:32.793339    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:32.795813 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:32.795772    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"471d349a-0799-4d4c-8e5d-e8a004868dd7","Type":"ContainerStarted","Data":"38a3ba8f7e02d305db408239768195d4c48ca9202ec923e88175bf45c55ddbd8"}
May 28 08:02:32.795813 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:32.795808    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"471d349a-0799-4d4c-8e5d-e8a004868dd7","Type":"ContainerStarted","Data":"4f55986079b7d4dc8b2a737874e54cf05a93d9e6ce87a7e51d1401fbe5f47924"}
May 28 08:02:32.795961 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:32.795822    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"471d349a-0799-4d4c-8e5d-e8a004868dd7","Type":"ContainerStarted","Data":"316c9467d9011143020b1f132b3e08da829a3f8a01d94fe05d609f982135e6aa"}
May 28 08:02:32.815146 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:32.815075    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj" podStartSLOduration=4.030645325 podStartE2EDuration="9.815062353s" podCreationTimestamp="2026-05-28 08:02:23 +0000 UTC" firstStartedPulling="2026-05-28 08:02:26.553556107 +0000 UTC m=+198.993628381" lastFinishedPulling="2026-05-28 08:02:32.337973124 +0000 UTC m=+204.778045409" observedRunningTime="2026-05-28 08:02:32.812370735 +0000 UTC m=+205.252443025" watchObservedRunningTime="2026-05-28 08:02:32.815062353 +0000 UTC m=+205.255134695"
May 28 08:02:33.804766 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:33.804731    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"471d349a-0799-4d4c-8e5d-e8a004868dd7","Type":"ContainerStarted","Data":"a6550277dcbe046b0f468526dc185b4f138f6c0b2319ac37e11d4b24218bbb83"}
May 28 08:02:33.804766 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:33.804771    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"471d349a-0799-4d4c-8e5d-e8a004868dd7","Type":"ContainerStarted","Data":"d7cc86a04c25d5c35b9f431ba009179f5c17edddf66a40f60bf3e3adfc0497e2"}
May 28 08:02:33.805322 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:33.804780    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"471d349a-0799-4d4c-8e5d-e8a004868dd7","Type":"ContainerStarted","Data":"3f7b27182f664275ef79f339cb4b48f4f24181693bb756a973e3b25785b3d3b2"}
May 28 08:02:33.830277 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:33.830226    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.037636175 podStartE2EDuration="11.830207349s" podCreationTimestamp="2026-05-28 08:02:22 +0000 UTC" firstStartedPulling="2026-05-28 08:02:26.548893568 +0000 UTC m=+198.988965850" lastFinishedPulling="2026-05-28 08:02:32.341464754 +0000 UTC m=+204.781537024" observedRunningTime="2026-05-28 08:02:33.828261677 +0000 UTC m=+206.268333994" watchObservedRunningTime="2026-05-28 08:02:33.830207349 +0000 UTC m=+206.270279638"
May 28 08:02:38.810755 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:38.810726    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6bb9978d9c-gn2xj"
May 28 08:02:54.797938 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:54.797880    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-596d4f4dc7-jl225" podUID="b50dcb32-5e7b-4087-a64d-7cc64aa1113c" containerName="console" containerID="cri-o://2f44c186bb9b933856c709fd3c8d29a02e1d5414917f83a612969142d5a1bc0a" gracePeriod=15
May 28 08:02:55.051031 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.050976    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-596d4f4dc7-jl225_b50dcb32-5e7b-4087-a64d-7cc64aa1113c/console/0.log"
May 28 08:02:55.051142 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.051048    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:55.142074 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.142039    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-oauth-config\") pod \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") "
May 28 08:02:55.142074 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.142079    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p826w\" (UniqueName: \"kubernetes.io/projected/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-kube-api-access-p826w\") pod \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") "
May 28 08:02:55.142315 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.142108    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-oauth-serving-cert\") pod \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") "
May 28 08:02:55.142315 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.142130    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-trusted-ca-bundle\") pod \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") "
May 28 08:02:55.142315 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.142148    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-config\") pod \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") "
May 28 08:02:55.142315 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.142217    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-service-ca\") pod \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") "
May 28 08:02:55.142315 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.142282    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-serving-cert\") pod \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\" (UID: \"b50dcb32-5e7b-4087-a64d-7cc64aa1113c\") "
May 28 08:02:55.142659 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.142614    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b50dcb32-5e7b-4087-a64d-7cc64aa1113c" (UID: "b50dcb32-5e7b-4087-a64d-7cc64aa1113c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 08:02:55.142745 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.142668    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b50dcb32-5e7b-4087-a64d-7cc64aa1113c" (UID: "b50dcb32-5e7b-4087-a64d-7cc64aa1113c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 08:02:55.142745 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.142719    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-service-ca" (OuterVolumeSpecName: "service-ca") pod "b50dcb32-5e7b-4087-a64d-7cc64aa1113c" (UID: "b50dcb32-5e7b-4087-a64d-7cc64aa1113c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 08:02:55.142745 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.142724    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-config" (OuterVolumeSpecName: "console-config") pod "b50dcb32-5e7b-4087-a64d-7cc64aa1113c" (UID: "b50dcb32-5e7b-4087-a64d-7cc64aa1113c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 08:02:55.144428 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.144402    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b50dcb32-5e7b-4087-a64d-7cc64aa1113c" (UID: "b50dcb32-5e7b-4087-a64d-7cc64aa1113c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:02:55.144527 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.144444    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-kube-api-access-p826w" (OuterVolumeSpecName: "kube-api-access-p826w") pod "b50dcb32-5e7b-4087-a64d-7cc64aa1113c" (UID: "b50dcb32-5e7b-4087-a64d-7cc64aa1113c"). InnerVolumeSpecName "kube-api-access-p826w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:02:55.144571 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.144531    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b50dcb32-5e7b-4087-a64d-7cc64aa1113c" (UID: "b50dcb32-5e7b-4087-a64d-7cc64aa1113c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:02:55.242992 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.242958    2551 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-service-ca\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:02:55.242992 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.242985    2551 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-serving-cert\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:02:55.242992 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.242996    2551 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-oauth-config\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:02:55.243194 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.243005    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p826w\" (UniqueName: \"kubernetes.io/projected/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-kube-api-access-p826w\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:02:55.243194 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.243014    2551 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-oauth-serving-cert\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:02:55.243194 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.243024    2551 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-trusted-ca-bundle\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:02:55.243194 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.243033    2551 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b50dcb32-5e7b-4087-a64d-7cc64aa1113c-console-config\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:02:55.870423 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.870397    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-596d4f4dc7-jl225_b50dcb32-5e7b-4087-a64d-7cc64aa1113c/console/0.log"
May 28 08:02:55.870829 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.870438    2551 generic.go:358] "Generic (PLEG): container finished" podID="b50dcb32-5e7b-4087-a64d-7cc64aa1113c" containerID="2f44c186bb9b933856c709fd3c8d29a02e1d5414917f83a612969142d5a1bc0a" exitCode=2
May 28 08:02:55.870829 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.870511    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596d4f4dc7-jl225"
May 28 08:02:55.870829 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.870530    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596d4f4dc7-jl225" event={"ID":"b50dcb32-5e7b-4087-a64d-7cc64aa1113c","Type":"ContainerDied","Data":"2f44c186bb9b933856c709fd3c8d29a02e1d5414917f83a612969142d5a1bc0a"}
May 28 08:02:55.870829 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.870571    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596d4f4dc7-jl225" event={"ID":"b50dcb32-5e7b-4087-a64d-7cc64aa1113c","Type":"ContainerDied","Data":"d517104c42a12576ec666fab905e850ef346cef9ba023570e5c21858349fc7dc"}
May 28 08:02:55.870829 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.870587    2551 scope.go:117] "RemoveContainer" containerID="2f44c186bb9b933856c709fd3c8d29a02e1d5414917f83a612969142d5a1bc0a"
May 28 08:02:55.878771 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.878753    2551 scope.go:117] "RemoveContainer" containerID="2f44c186bb9b933856c709fd3c8d29a02e1d5414917f83a612969142d5a1bc0a"
May 28 08:02:55.879026 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:02:55.879007    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f44c186bb9b933856c709fd3c8d29a02e1d5414917f83a612969142d5a1bc0a\": container with ID starting with 2f44c186bb9b933856c709fd3c8d29a02e1d5414917f83a612969142d5a1bc0a not found: ID does not exist" containerID="2f44c186bb9b933856c709fd3c8d29a02e1d5414917f83a612969142d5a1bc0a"
May 28 08:02:55.879085 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.879033    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f44c186bb9b933856c709fd3c8d29a02e1d5414917f83a612969142d5a1bc0a"} err="failed to get container status \"2f44c186bb9b933856c709fd3c8d29a02e1d5414917f83a612969142d5a1bc0a\": rpc error: code = NotFound desc = could not find container \"2f44c186bb9b933856c709fd3c8d29a02e1d5414917f83a612969142d5a1bc0a\": container with ID starting with 2f44c186bb9b933856c709fd3c8d29a02e1d5414917f83a612969142d5a1bc0a not found: ID does not exist"
May 28 08:02:55.891764 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.891738    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-596d4f4dc7-jl225"]
May 28 08:02:55.895154 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:55.895133    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-596d4f4dc7-jl225"]
May 28 08:02:56.194417 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:56.194333    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b50dcb32-5e7b-4087-a64d-7cc64aa1113c" path="/var/lib/kubelet/pods/b50dcb32-5e7b-4087-a64d-7cc64aa1113c/volumes"
May 28 08:02:59.548759 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:02:59.548732    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8vcnc_91168ae0-b0ae-482d-be58-f879dcae55f1/dns-node-resolver/0.log"
May 28 08:03:00.149336 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:00.149311    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56dd9859f6-76gdf_0039c6a2-d3a2-40aa-9799-8237db852588/router/0.log"
May 28 08:03:18.948512 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:18.948474    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs\") pod \"network-metrics-daemon-dfswh\" (UID: \"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd\") " pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 08:03:18.950675 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:18.950655    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd-metrics-certs\") pod \"network-metrics-daemon-dfswh\" (UID: \"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd\") " pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 08:03:18.994058 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:18.994033    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q9jcs\""
May 28 08:03:19.002719 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:19.002693    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfswh"
May 28 08:03:19.123426 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:19.123267    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dfswh"]
May 28 08:03:19.125978 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:03:19.125952    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f629a2a_caeb_4e3c_8dfb_b835ecd14fbd.slice/crio-065ea9e7cae5b24669bc70b6de5fe0b025b18cf28fb3ec09cc5cc2fb9009eb4b WatchSource:0}: Error finding container 065ea9e7cae5b24669bc70b6de5fe0b025b18cf28fb3ec09cc5cc2fb9009eb4b: Status 404 returned error can't find the container with id 065ea9e7cae5b24669bc70b6de5fe0b025b18cf28fb3ec09cc5cc2fb9009eb4b
May 28 08:03:19.937241 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:19.937202    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dfswh" event={"ID":"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd","Type":"ContainerStarted","Data":"065ea9e7cae5b24669bc70b6de5fe0b025b18cf28fb3ec09cc5cc2fb9009eb4b"}
May 28 08:03:20.943364 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:20.943320    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dfswh" event={"ID":"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd","Type":"ContainerStarted","Data":"42730edc4a0a81f78a9718ed16cb97ac4dccb77649f89393c59c1616adfe0763"}
May 28 08:03:21.950238 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:21.950203    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dfswh" event={"ID":"2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd","Type":"ContainerStarted","Data":"dd7309c03ccc4f49e77f12afbcb32e3e3a4a4fe03256b6bbf10ea65cf7fd409a"}
May 28 08:03:21.964683 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:21.964418    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dfswh" podStartSLOduration=252.260426417 podStartE2EDuration="4m13.964402972s" podCreationTimestamp="2026-05-28 07:59:08 +0000 UTC" firstStartedPulling="2026-05-28 08:03:19.127819459 +0000 UTC m=+251.567891725" lastFinishedPulling="2026-05-28 08:03:20.83179601 +0000 UTC m=+253.271868280" observedRunningTime="2026-05-28 08:03:21.964252818 +0000 UTC m=+254.404325108" watchObservedRunningTime="2026-05-28 08:03:21.964402972 +0000 UTC m=+254.404475255"
May 28 08:03:41.880710 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:41.880667    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 08:03:41.881249 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:41.881124    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="alertmanager" containerID="cri-o://316c9467d9011143020b1f132b3e08da829a3f8a01d94fe05d609f982135e6aa" gracePeriod=120
May 28 08:03:41.881249 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:41.881200    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="kube-rbac-proxy-metric" containerID="cri-o://d7cc86a04c25d5c35b9f431ba009179f5c17edddf66a40f60bf3e3adfc0497e2" gracePeriod=120
May 28 08:03:41.881362 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:41.881249    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="config-reloader" containerID="cri-o://4f55986079b7d4dc8b2a737874e54cf05a93d9e6ce87a7e51d1401fbe5f47924" gracePeriod=120
May 28 08:03:41.881362 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:41.881257    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="kube-rbac-proxy" containerID="cri-o://3f7b27182f664275ef79f339cb4b48f4f24181693bb756a973e3b25785b3d3b2" gracePeriod=120
May 28 08:03:41.881362 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:41.881234    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="kube-rbac-proxy-web" containerID="cri-o://38a3ba8f7e02d305db408239768195d4c48ca9202ec923e88175bf45c55ddbd8" gracePeriod=120
May 28 08:03:41.881362 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:41.881254    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="prom-label-proxy" containerID="cri-o://a6550277dcbe046b0f468526dc185b4f138f6c0b2319ac37e11d4b24218bbb83" gracePeriod=120
May 28 08:03:42.009829 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:42.009794    2551 generic.go:358] "Generic (PLEG): container finished" podID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerID="a6550277dcbe046b0f468526dc185b4f138f6c0b2319ac37e11d4b24218bbb83" exitCode=0
May 28 08:03:42.009829 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:42.009820    2551 generic.go:358] "Generic (PLEG): container finished" podID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerID="d7cc86a04c25d5c35b9f431ba009179f5c17edddf66a40f60bf3e3adfc0497e2" exitCode=0
May 28 08:03:42.009829 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:42.009826    2551 generic.go:358] "Generic (PLEG): container finished" podID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerID="3f7b27182f664275ef79f339cb4b48f4f24181693bb756a973e3b25785b3d3b2" exitCode=0
May 28 08:03:42.009829 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:42.009832    2551 generic.go:358] "Generic (PLEG): container finished" podID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerID="4f55986079b7d4dc8b2a737874e54cf05a93d9e6ce87a7e51d1401fbe5f47924" exitCode=0
May 28 08:03:42.010028 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:42.009838    2551 generic.go:358] "Generic (PLEG): container finished" podID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerID="316c9467d9011143020b1f132b3e08da829a3f8a01d94fe05d609f982135e6aa" exitCode=0
May 28 08:03:42.010028 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:42.009863    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"471d349a-0799-4d4c-8e5d-e8a004868dd7","Type":"ContainerDied","Data":"a6550277dcbe046b0f468526dc185b4f138f6c0b2319ac37e11d4b24218bbb83"}
May 28 08:03:42.010028 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:42.009895    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"471d349a-0799-4d4c-8e5d-e8a004868dd7","Type":"ContainerDied","Data":"d7cc86a04c25d5c35b9f431ba009179f5c17edddf66a40f60bf3e3adfc0497e2"}
May 28 08:03:42.010028 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:42.009905    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"471d349a-0799-4d4c-8e5d-e8a004868dd7","Type":"ContainerDied","Data":"3f7b27182f664275ef79f339cb4b48f4f24181693bb756a973e3b25785b3d3b2"}
May 28 08:03:42.010028 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:42.009913    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"471d349a-0799-4d4c-8e5d-e8a004868dd7","Type":"ContainerDied","Data":"4f55986079b7d4dc8b2a737874e54cf05a93d9e6ce87a7e51d1401fbe5f47924"}
May 28 08:03:42.010028 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:42.009921    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"471d349a-0799-4d4c-8e5d-e8a004868dd7","Type":"ContainerDied","Data":"316c9467d9011143020b1f132b3e08da829a3f8a01d94fe05d609f982135e6aa"}
May 28 08:03:43.229746 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.229719    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:43.354917 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.354831    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-config-volume\") pod \"471d349a-0799-4d4c-8e5d-e8a004868dd7\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") "
May 28 08:03:43.354917 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.354874    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"471d349a-0799-4d4c-8e5d-e8a004868dd7\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") "
May 28 08:03:43.354917 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.354901    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/471d349a-0799-4d4c-8e5d-e8a004868dd7-metrics-client-ca\") pod \"471d349a-0799-4d4c-8e5d-e8a004868dd7\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") "
May 28 08:03:43.355173 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.354956    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-main-tls\") pod \"471d349a-0799-4d4c-8e5d-e8a004868dd7\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") "
May 28 08:03:43.355173 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.354983    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-web-config\") pod \"471d349a-0799-4d4c-8e5d-e8a004868dd7\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") "
May 28 08:03:43.355173 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.355007    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlq9t\" (UniqueName: \"kubernetes.io/projected/471d349a-0799-4d4c-8e5d-e8a004868dd7-kube-api-access-tlq9t\") pod \"471d349a-0799-4d4c-8e5d-e8a004868dd7\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") "
May 28 08:03:43.355173 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.355097    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/471d349a-0799-4d4c-8e5d-e8a004868dd7-alertmanager-main-db\") pod \"471d349a-0799-4d4c-8e5d-e8a004868dd7\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") "
May 28 08:03:43.355173 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.355141    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/471d349a-0799-4d4c-8e5d-e8a004868dd7-alertmanager-trusted-ca-bundle\") pod \"471d349a-0799-4d4c-8e5d-e8a004868dd7\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") "
May 28 08:03:43.355466 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.355188    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/471d349a-0799-4d4c-8e5d-e8a004868dd7-config-out\") pod \"471d349a-0799-4d4c-8e5d-e8a004868dd7\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") "
May 28 08:03:43.355466 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.355223    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-cluster-tls-config\") pod \"471d349a-0799-4d4c-8e5d-e8a004868dd7\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") "
May 28 08:03:43.355466 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.355258    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy-web\") pod \"471d349a-0799-4d4c-8e5d-e8a004868dd7\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") "
May 28 08:03:43.355466 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.355293    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy\") pod \"471d349a-0799-4d4c-8e5d-e8a004868dd7\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") "
May 28 08:03:43.355466 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.355378    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/471d349a-0799-4d4c-8e5d-e8a004868dd7-tls-assets\") pod \"471d349a-0799-4d4c-8e5d-e8a004868dd7\" (UID: \"471d349a-0799-4d4c-8e5d-e8a004868dd7\") "
May 28 08:03:43.355466 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.355367    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/471d349a-0799-4d4c-8e5d-e8a004868dd7-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "471d349a-0799-4d4c-8e5d-e8a004868dd7" (UID: "471d349a-0799-4d4c-8e5d-e8a004868dd7"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 08:03:43.355755 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.355690    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/471d349a-0799-4d4c-8e5d-e8a004868dd7-metrics-client-ca\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:03:43.355755 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.355715    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471d349a-0799-4d4c-8e5d-e8a004868dd7-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "471d349a-0799-4d4c-8e5d-e8a004868dd7" (UID: "471d349a-0799-4d4c-8e5d-e8a004868dd7"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 08:03:43.357763 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.357740    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-config-volume" (OuterVolumeSpecName: "config-volume") pod "471d349a-0799-4d4c-8e5d-e8a004868dd7" (UID: "471d349a-0799-4d4c-8e5d-e8a004868dd7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:03:43.358024 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.357995    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "471d349a-0799-4d4c-8e5d-e8a004868dd7" (UID: "471d349a-0799-4d4c-8e5d-e8a004868dd7"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:03:43.358474 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.358319    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/471d349a-0799-4d4c-8e5d-e8a004868dd7-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "471d349a-0799-4d4c-8e5d-e8a004868dd7" (UID: "471d349a-0799-4d4c-8e5d-e8a004868dd7"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 08:03:43.358474 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.358424    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "471d349a-0799-4d4c-8e5d-e8a004868dd7" (UID: "471d349a-0799-4d4c-8e5d-e8a004868dd7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:03:43.358474 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.358434    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471d349a-0799-4d4c-8e5d-e8a004868dd7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "471d349a-0799-4d4c-8e5d-e8a004868dd7" (UID: "471d349a-0799-4d4c-8e5d-e8a004868dd7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:03:43.358474 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.358447    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471d349a-0799-4d4c-8e5d-e8a004868dd7-kube-api-access-tlq9t" (OuterVolumeSpecName: "kube-api-access-tlq9t") pod "471d349a-0799-4d4c-8e5d-e8a004868dd7" (UID: "471d349a-0799-4d4c-8e5d-e8a004868dd7"). InnerVolumeSpecName "kube-api-access-tlq9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:03:43.358736 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.358679    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471d349a-0799-4d4c-8e5d-e8a004868dd7-config-out" (OuterVolumeSpecName: "config-out") pod "471d349a-0799-4d4c-8e5d-e8a004868dd7" (UID: "471d349a-0799-4d4c-8e5d-e8a004868dd7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 08:03:43.358736 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.358708    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "471d349a-0799-4d4c-8e5d-e8a004868dd7" (UID: "471d349a-0799-4d4c-8e5d-e8a004868dd7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:03:43.359975 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.359953    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "471d349a-0799-4d4c-8e5d-e8a004868dd7" (UID: "471d349a-0799-4d4c-8e5d-e8a004868dd7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:03:43.363087 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.362989    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "471d349a-0799-4d4c-8e5d-e8a004868dd7" (UID: "471d349a-0799-4d4c-8e5d-e8a004868dd7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:03:43.368999 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.368968    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-web-config" (OuterVolumeSpecName: "web-config") pod "471d349a-0799-4d4c-8e5d-e8a004868dd7" (UID: "471d349a-0799-4d4c-8e5d-e8a004868dd7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:03:43.456689 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.456645    2551 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/471d349a-0799-4d4c-8e5d-e8a004868dd7-tls-assets\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:03:43.456689 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.456680    2551 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-config-volume\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:03:43.456689 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.456691    2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:03:43.456909 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.456702    2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-main-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:03:43.456909 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.456713    2551 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-web-config\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:03:43.456909 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.456722    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlq9t\" (UniqueName: \"kubernetes.io/projected/471d349a-0799-4d4c-8e5d-e8a004868dd7-kube-api-access-tlq9t\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:03:43.456909 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.456732    2551 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/471d349a-0799-4d4c-8e5d-e8a004868dd7-alertmanager-main-db\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:03:43.456909 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.456741    2551 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/471d349a-0799-4d4c-8e5d-e8a004868dd7-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:03:43.456909 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.456751    2551 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/471d349a-0799-4d4c-8e5d-e8a004868dd7-config-out\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:03:43.456909 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.456759    2551 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-cluster-tls-config\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:03:43.456909 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.456767    2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:03:43.456909 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:43.456776    2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/471d349a-0799-4d4c-8e5d-e8a004868dd7-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:03:44.017846 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.017814    2551 generic.go:358] "Generic (PLEG): container finished" podID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerID="38a3ba8f7e02d305db408239768195d4c48ca9202ec923e88175bf45c55ddbd8" exitCode=0
May 28 08:03:44.017968 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.017876    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"471d349a-0799-4d4c-8e5d-e8a004868dd7","Type":"ContainerDied","Data":"38a3ba8f7e02d305db408239768195d4c48ca9202ec923e88175bf45c55ddbd8"}
May 28 08:03:44.017968 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.017904    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"471d349a-0799-4d4c-8e5d-e8a004868dd7","Type":"ContainerDied","Data":"c4943f7fefd5c6e58ebdad64104948b3948c6fa97fd5a50afcfde3e9d90817fb"}
May 28 08:03:44.017968 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.017920    2551 scope.go:117] "RemoveContainer" containerID="a6550277dcbe046b0f468526dc185b4f138f6c0b2319ac37e11d4b24218bbb83"
May 28 08:03:44.017968 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.017955    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.025047 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.025030    2551 scope.go:117] "RemoveContainer" containerID="d7cc86a04c25d5c35b9f431ba009179f5c17edddf66a40f60bf3e3adfc0497e2"
May 28 08:03:44.031863 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.031847    2551 scope.go:117] "RemoveContainer" containerID="3f7b27182f664275ef79f339cb4b48f4f24181693bb756a973e3b25785b3d3b2"
May 28 08:03:44.038134 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.038073    2551 scope.go:117] "RemoveContainer" containerID="38a3ba8f7e02d305db408239768195d4c48ca9202ec923e88175bf45c55ddbd8"
May 28 08:03:44.039652 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.039631    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 08:03:44.044789 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.044769    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 08:03:44.046783 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.046702    2551 scope.go:117] "RemoveContainer" containerID="4f55986079b7d4dc8b2a737874e54cf05a93d9e6ce87a7e51d1401fbe5f47924"
May 28 08:03:44.053210 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.053193    2551 scope.go:117] "RemoveContainer" containerID="316c9467d9011143020b1f132b3e08da829a3f8a01d94fe05d609f982135e6aa"
May 28 08:03:44.059912 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.059895    2551 scope.go:117] "RemoveContainer" containerID="259f2b12a51d5039249a168bcc614de7c89b0487dce91f3554423e82eb3ab5cb"
May 28 08:03:44.066244 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.066226    2551 scope.go:117] "RemoveContainer" containerID="a6550277dcbe046b0f468526dc185b4f138f6c0b2319ac37e11d4b24218bbb83"
May 28 08:03:44.066572 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:03:44.066552    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6550277dcbe046b0f468526dc185b4f138f6c0b2319ac37e11d4b24218bbb83\": container with ID starting with a6550277dcbe046b0f468526dc185b4f138f6c0b2319ac37e11d4b24218bbb83 not found: ID does not exist" containerID="a6550277dcbe046b0f468526dc185b4f138f6c0b2319ac37e11d4b24218bbb83"
May 28 08:03:44.066652 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.066581    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6550277dcbe046b0f468526dc185b4f138f6c0b2319ac37e11d4b24218bbb83"} err="failed to get container status \"a6550277dcbe046b0f468526dc185b4f138f6c0b2319ac37e11d4b24218bbb83\": rpc error: code = NotFound desc = could not find container \"a6550277dcbe046b0f468526dc185b4f138f6c0b2319ac37e11d4b24218bbb83\": container with ID starting with a6550277dcbe046b0f468526dc185b4f138f6c0b2319ac37e11d4b24218bbb83 not found: ID does not exist"
May 28 08:03:44.066652 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.066602    2551 scope.go:117] "RemoveContainer" containerID="d7cc86a04c25d5c35b9f431ba009179f5c17edddf66a40f60bf3e3adfc0497e2"
May 28 08:03:44.066882 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:03:44.066857    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7cc86a04c25d5c35b9f431ba009179f5c17edddf66a40f60bf3e3adfc0497e2\": container with ID starting with d7cc86a04c25d5c35b9f431ba009179f5c17edddf66a40f60bf3e3adfc0497e2 not found: ID does not exist" containerID="d7cc86a04c25d5c35b9f431ba009179f5c17edddf66a40f60bf3e3adfc0497e2"
May 28 08:03:44.066984 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.066890    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7cc86a04c25d5c35b9f431ba009179f5c17edddf66a40f60bf3e3adfc0497e2"} err="failed to get container status \"d7cc86a04c25d5c35b9f431ba009179f5c17edddf66a40f60bf3e3adfc0497e2\": rpc error: code = NotFound desc = could not find container \"d7cc86a04c25d5c35b9f431ba009179f5c17edddf66a40f60bf3e3adfc0497e2\": container with ID starting with d7cc86a04c25d5c35b9f431ba009179f5c17edddf66a40f60bf3e3adfc0497e2 not found: ID does not exist"
May 28 08:03:44.066984 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.066914    2551 scope.go:117] "RemoveContainer" containerID="3f7b27182f664275ef79f339cb4b48f4f24181693bb756a973e3b25785b3d3b2"
May 28 08:03:44.067242 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:03:44.067226    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7b27182f664275ef79f339cb4b48f4f24181693bb756a973e3b25785b3d3b2\": container with ID starting with 3f7b27182f664275ef79f339cb4b48f4f24181693bb756a973e3b25785b3d3b2 not found: ID does not exist" containerID="3f7b27182f664275ef79f339cb4b48f4f24181693bb756a973e3b25785b3d3b2"
May 28 08:03:44.067311 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.067247    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7b27182f664275ef79f339cb4b48f4f24181693bb756a973e3b25785b3d3b2"} err="failed to get container status \"3f7b27182f664275ef79f339cb4b48f4f24181693bb756a973e3b25785b3d3b2\": rpc error: code = NotFound desc = could not find container \"3f7b27182f664275ef79f339cb4b48f4f24181693bb756a973e3b25785b3d3b2\": container with ID starting with 3f7b27182f664275ef79f339cb4b48f4f24181693bb756a973e3b25785b3d3b2 not found: ID does not exist"
May 28 08:03:44.067311 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.067263    2551 scope.go:117] "RemoveContainer" containerID="38a3ba8f7e02d305db408239768195d4c48ca9202ec923e88175bf45c55ddbd8"
May 28 08:03:44.067911 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:03:44.067888    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a3ba8f7e02d305db408239768195d4c48ca9202ec923e88175bf45c55ddbd8\": container with ID starting with 38a3ba8f7e02d305db408239768195d4c48ca9202ec923e88175bf45c55ddbd8 not found: ID does not exist" containerID="38a3ba8f7e02d305db408239768195d4c48ca9202ec923e88175bf45c55ddbd8"
May 28 08:03:44.067978 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.067917    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a3ba8f7e02d305db408239768195d4c48ca9202ec923e88175bf45c55ddbd8"} err="failed to get container status \"38a3ba8f7e02d305db408239768195d4c48ca9202ec923e88175bf45c55ddbd8\": rpc error: code = NotFound desc = could not find container \"38a3ba8f7e02d305db408239768195d4c48ca9202ec923e88175bf45c55ddbd8\": container with ID starting with 38a3ba8f7e02d305db408239768195d4c48ca9202ec923e88175bf45c55ddbd8 not found: ID does not exist"
May 28 08:03:44.067978 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.067932    2551 scope.go:117] "RemoveContainer" containerID="4f55986079b7d4dc8b2a737874e54cf05a93d9e6ce87a7e51d1401fbe5f47924"
May 28 08:03:44.067978 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.067951    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 08:03:44.068185 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:03:44.068164    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f55986079b7d4dc8b2a737874e54cf05a93d9e6ce87a7e51d1401fbe5f47924\": container with ID starting with 4f55986079b7d4dc8b2a737874e54cf05a93d9e6ce87a7e51d1401fbe5f47924 not found: ID does not exist" containerID="4f55986079b7d4dc8b2a737874e54cf05a93d9e6ce87a7e51d1401fbe5f47924"
May 28 08:03:44.068278 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068194    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f55986079b7d4dc8b2a737874e54cf05a93d9e6ce87a7e51d1401fbe5f47924"} err="failed to get container status \"4f55986079b7d4dc8b2a737874e54cf05a93d9e6ce87a7e51d1401fbe5f47924\": rpc error: code = NotFound desc = could not find container \"4f55986079b7d4dc8b2a737874e54cf05a93d9e6ce87a7e51d1401fbe5f47924\": container with ID starting with 4f55986079b7d4dc8b2a737874e54cf05a93d9e6ce87a7e51d1401fbe5f47924 not found: ID does not exist"
May 28 08:03:44.068278 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068219    2551 scope.go:117] "RemoveContainer" containerID="316c9467d9011143020b1f132b3e08da829a3f8a01d94fe05d609f982135e6aa"
May 28 08:03:44.068415 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068300    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="config-reloader"
May 28 08:03:44.068415 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068313    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="config-reloader"
May 28 08:03:44.068415 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068323    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="prom-label-proxy"
May 28 08:03:44.068415 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068330    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="prom-label-proxy"
May 28 08:03:44.068415 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068345    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b50dcb32-5e7b-4087-a64d-7cc64aa1113c" containerName="console"
May 28 08:03:44.068415 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068353    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b50dcb32-5e7b-4087-a64d-7cc64aa1113c" containerName="console"
May 28 08:03:44.068415 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068360    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="kube-rbac-proxy"
May 28 08:03:44.068415 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068365    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="kube-rbac-proxy"
May 28 08:03:44.068415 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068371    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="kube-rbac-proxy-metric"
May 28 08:03:44.068415 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068378    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="kube-rbac-proxy-metric"
May 28 08:03:44.068415 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068408    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="init-config-reloader"
May 28 08:03:44.068415 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068416    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="init-config-reloader"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068428    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="kube-rbac-proxy-web"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068435    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="kube-rbac-proxy-web"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068451    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="alertmanager"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068458    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="alertmanager"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:03:44.068474    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316c9467d9011143020b1f132b3e08da829a3f8a01d94fe05d609f982135e6aa\": container with ID starting with 316c9467d9011143020b1f132b3e08da829a3f8a01d94fe05d609f982135e6aa not found: ID does not exist" containerID="316c9467d9011143020b1f132b3e08da829a3f8a01d94fe05d609f982135e6aa"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068501    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316c9467d9011143020b1f132b3e08da829a3f8a01d94fe05d609f982135e6aa"} err="failed to get container status \"316c9467d9011143020b1f132b3e08da829a3f8a01d94fe05d609f982135e6aa\": rpc error: code = NotFound desc = could not find container \"316c9467d9011143020b1f132b3e08da829a3f8a01d94fe05d609f982135e6aa\": container with ID starting with 316c9467d9011143020b1f132b3e08da829a3f8a01d94fe05d609f982135e6aa not found: ID does not exist"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068522    2551 scope.go:117] "RemoveContainer" containerID="259f2b12a51d5039249a168bcc614de7c89b0487dce91f3554423e82eb3ab5cb"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068536    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="alertmanager"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068547    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="kube-rbac-proxy-web"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068554    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="kube-rbac-proxy-metric"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068562    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="b50dcb32-5e7b-4087-a64d-7cc64aa1113c" containerName="console"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068569    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="config-reloader"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068577    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="prom-label-proxy"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068584    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" containerName="kube-rbac-proxy"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:03:44.068774    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"259f2b12a51d5039249a168bcc614de7c89b0487dce91f3554423e82eb3ab5cb\": container with ID starting with 259f2b12a51d5039249a168bcc614de7c89b0487dce91f3554423e82eb3ab5cb not found: ID does not exist" containerID="259f2b12a51d5039249a168bcc614de7c89b0487dce91f3554423e82eb3ab5cb"
May 28 08:03:44.068900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.068795    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259f2b12a51d5039249a168bcc614de7c89b0487dce91f3554423e82eb3ab5cb"} err="failed to get container status \"259f2b12a51d5039249a168bcc614de7c89b0487dce91f3554423e82eb3ab5cb\": rpc error: code = NotFound desc = could not find container \"259f2b12a51d5039249a168bcc614de7c89b0487dce91f3554423e82eb3ab5cb\": container with ID starting with 259f2b12a51d5039249a168bcc614de7c89b0487dce91f3554423e82eb3ab5cb not found: ID does not exist"
May 28 08:03:44.073795 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.073779    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.076173 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.076155    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 28 08:03:44.076173 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.076170    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 28 08:03:44.076309 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.076205    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 28 08:03:44.076309 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.076222    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 28 08:03:44.076309 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.076169    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 28 08:03:44.076600 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.076585    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 28 08:03:44.076711 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.076697    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xlk5w\""
May 28 08:03:44.076980 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.076958    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 28 08:03:44.077081 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.076979    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 28 08:03:44.081521 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.081504    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 28 08:03:44.084598 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.084580    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 08:03:44.162745 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.162712    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a17ff24-aaf1-4538-b12e-84eb498faff3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.162912 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.162783    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.162912 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.162805    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.162912 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.162840    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a17ff24-aaf1-4538-b12e-84eb498faff3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.162912 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.162868    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.162912 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.162904    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a17ff24-aaf1-4538-b12e-84eb498faff3-config-out\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.163158 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.162930    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.163158 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.162950    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a17ff24-aaf1-4538-b12e-84eb498faff3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.163158 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.162968    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-config-volume\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.163158 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.162991    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.163158 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.163055    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-web-config\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.163158 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.163113    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2a17ff24-aaf1-4538-b12e-84eb498faff3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.163158 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.163133    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhf5n\" (UniqueName: \"kubernetes.io/projected/2a17ff24-aaf1-4538-b12e-84eb498faff3-kube-api-access-hhf5n\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.188143 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.188115    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-86d69c8dfd-ldf5r"]
May 28 08:03:44.191536 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.191483    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.194353 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.193950    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 28 08:03:44.194353 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.194017    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 28 08:03:44.194353 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.194102    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 28 08:03:44.194353 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.194128    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dfc4r\""
May 28 08:03:44.194353 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.194225    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 28 08:03:44.194353 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.194244    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 28 08:03:44.197002 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.196978    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="471d349a-0799-4d4c-8e5d-e8a004868dd7" path="/var/lib/kubelet/pods/471d349a-0799-4d4c-8e5d-e8a004868dd7/volumes"
May 28 08:03:44.198780 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.198761    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 28 08:03:44.203486 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.203378    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86d69c8dfd-ldf5r"]
May 28 08:03:44.264313 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264237    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2a17ff24-aaf1-4538-b12e-84eb498faff3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.264313 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264269    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hhf5n\" (UniqueName: \"kubernetes.io/projected/2a17ff24-aaf1-4538-b12e-84eb498faff3-kube-api-access-hhf5n\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.264313 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264292    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-trusted-ca-bundle\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.264802 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264325    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a17ff24-aaf1-4538-b12e-84eb498faff3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.264802 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264424    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.264802 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264460    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.264802 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264492    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f485b61-7853-4421-8b61-e02a02152a65-console-oauth-config\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.264802 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264528    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a17ff24-aaf1-4538-b12e-84eb498faff3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.264802 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264554    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.264802 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264586    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a17ff24-aaf1-4538-b12e-84eb498faff3-config-out\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.264802 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264625    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.264802 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264650    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a17ff24-aaf1-4538-b12e-84eb498faff3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.264802 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264689    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-config-volume\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.264802 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264704    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2a17ff24-aaf1-4538-b12e-84eb498faff3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.264802 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264716    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-oauth-serving-cert\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.264802 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264766    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.264802 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264797    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmztw\" (UniqueName: \"kubernetes.io/projected/8f485b61-7853-4421-8b61-e02a02152a65-kube-api-access-wmztw\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.265498 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264829    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f485b61-7853-4421-8b61-e02a02152a65-console-serving-cert\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.265498 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264872    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-web-config\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.265498 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264910    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-service-ca\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.265498 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.264958    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-console-config\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.265498 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.265076    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a17ff24-aaf1-4538-b12e-84eb498faff3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.266195 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.266168    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a17ff24-aaf1-4538-b12e-84eb498faff3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.267397 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.267350    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a17ff24-aaf1-4538-b12e-84eb498faff3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.267514 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.267489    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.267615 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.267593    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.267708 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.267688    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-web-config\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.267957 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.267930    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.268050 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.268034    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.268678 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.268656    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.268754 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.268705    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a17ff24-aaf1-4538-b12e-84eb498faff3-config-out\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.269460 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.269444    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2a17ff24-aaf1-4538-b12e-84eb498faff3-config-volume\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.273053 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.273031    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhf5n\" (UniqueName: \"kubernetes.io/projected/2a17ff24-aaf1-4538-b12e-84eb498faff3-kube-api-access-hhf5n\") pod \"alertmanager-main-0\" (UID: \"2a17ff24-aaf1-4538-b12e-84eb498faff3\") " pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.365743 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.365714    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-service-ca\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.365899 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.365760    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-console-config\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.365899 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.365803    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-trusted-ca-bundle\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.365899 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.365853    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f485b61-7853-4421-8b61-e02a02152a65-console-oauth-config\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.365899 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.365893    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-oauth-serving-cert\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.366067 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.365924    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wmztw\" (UniqueName: \"kubernetes.io/projected/8f485b61-7853-4421-8b61-e02a02152a65-kube-api-access-wmztw\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.366067 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.365949    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f485b61-7853-4421-8b61-e02a02152a65-console-serving-cert\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.366539 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.366514    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-console-config\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.366539 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.366532    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-service-ca\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.366689 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.366650    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-oauth-serving-cert\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.366755 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.366689    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-trusted-ca-bundle\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.368189 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.368169    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f485b61-7853-4421-8b61-e02a02152a65-console-oauth-config\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.368366 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.368350    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f485b61-7853-4421-8b61-e02a02152a65-console-serving-cert\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.372974 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.372955    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmztw\" (UniqueName: \"kubernetes.io/projected/8f485b61-7853-4421-8b61-e02a02152a65-kube-api-access-wmztw\") pod \"console-86d69c8dfd-ldf5r\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") " pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.382892 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.382867    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 28 08:03:44.502072 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.502037    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:44.511490 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.511454    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 28 08:03:44.514358 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:03:44.514291    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a17ff24_aaf1_4538_b12e_84eb498faff3.slice/crio-623f4ac5e7c538fb946a422dc4c1a90bfe53939838190904d44977eabc964cf7 WatchSource:0}: Error finding container 623f4ac5e7c538fb946a422dc4c1a90bfe53939838190904d44977eabc964cf7: Status 404 returned error can't find the container with id 623f4ac5e7c538fb946a422dc4c1a90bfe53939838190904d44977eabc964cf7
May 28 08:03:44.647677 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:44.647534    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86d69c8dfd-ldf5r"]
May 28 08:03:44.658317 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:03:44.658284    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f485b61_7853_4421_8b61_e02a02152a65.slice/crio-e1d5f0eced7bd1f992d3adf0ef2a4c3f570d3e63b62b084ed176b9daa69ce067 WatchSource:0}: Error finding container e1d5f0eced7bd1f992d3adf0ef2a4c3f570d3e63b62b084ed176b9daa69ce067: Status 404 returned error can't find the container with id e1d5f0eced7bd1f992d3adf0ef2a4c3f570d3e63b62b084ed176b9daa69ce067
May 28 08:03:45.022221 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:45.022184    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d69c8dfd-ldf5r" event={"ID":"8f485b61-7853-4421-8b61-e02a02152a65","Type":"ContainerStarted","Data":"302eee53ac233d2635ec4a60af7323fc2cf5c7059cbd68ff6deecc545ce1205a"}
May 28 08:03:45.022221 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:45.022221    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d69c8dfd-ldf5r" event={"ID":"8f485b61-7853-4421-8b61-e02a02152a65","Type":"ContainerStarted","Data":"e1d5f0eced7bd1f992d3adf0ef2a4c3f570d3e63b62b084ed176b9daa69ce067"}
May 28 08:03:45.023496 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:45.023473    2551 generic.go:358] "Generic (PLEG): container finished" podID="2a17ff24-aaf1-4538-b12e-84eb498faff3" containerID="7482b08555e577509e2322fbf9daa8a1025e0c652d48984194218fa2abf68e26" exitCode=0
May 28 08:03:45.023579 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:45.023545    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a17ff24-aaf1-4538-b12e-84eb498faff3","Type":"ContainerDied","Data":"7482b08555e577509e2322fbf9daa8a1025e0c652d48984194218fa2abf68e26"}
May 28 08:03:45.023579 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:45.023563    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a17ff24-aaf1-4538-b12e-84eb498faff3","Type":"ContainerStarted","Data":"623f4ac5e7c538fb946a422dc4c1a90bfe53939838190904d44977eabc964cf7"}
May 28 08:03:45.039169 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:45.038086    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-86d69c8dfd-ldf5r" podStartSLOduration=1.038069334 podStartE2EDuration="1.038069334s" podCreationTimestamp="2026-05-28 08:03:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:03:45.037538464 +0000 UTC m=+277.477610755" watchObservedRunningTime="2026-05-28 08:03:45.038069334 +0000 UTC m=+277.478141624"
May 28 08:03:46.032455 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:46.032414    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a17ff24-aaf1-4538-b12e-84eb498faff3","Type":"ContainerStarted","Data":"41c4895d9ac4f5b9a84022d5fac4caf6d1afd336e5953bfdecf8ec1c630b0872"}
May 28 08:03:46.032455 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:46.032460    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a17ff24-aaf1-4538-b12e-84eb498faff3","Type":"ContainerStarted","Data":"727ae99155cbcb1dc94582434fcbe4a5d0434c93d48712e709c1202dfb42043e"}
May 28 08:03:46.032879 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:46.032473    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a17ff24-aaf1-4538-b12e-84eb498faff3","Type":"ContainerStarted","Data":"3e684f37fa9bf29b7b35ebc5d750118b4c71e640ed6c225af0bf1a97756502de"}
May 28 08:03:46.032879 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:46.032484    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a17ff24-aaf1-4538-b12e-84eb498faff3","Type":"ContainerStarted","Data":"e8af1f3a34fb715b3ffa11767b2b163fd2f5b2b5ac6895034a283c41246f2770"}
May 28 08:03:46.032879 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:46.032495    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a17ff24-aaf1-4538-b12e-84eb498faff3","Type":"ContainerStarted","Data":"fb46d8cbe7b53e6908a1ff5ef5131e4b941c264468350cbd59190f535329ce6b"}
May 28 08:03:46.032879 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:46.032506    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2a17ff24-aaf1-4538-b12e-84eb498faff3","Type":"ContainerStarted","Data":"8a60fcd6c4b5bb3bd09bc2bdab28dfcd0649afd533c67ce2598c14387b2e9503"}
May 28 08:03:46.059196 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:46.059136    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.059120146 podStartE2EDuration="2.059120146s" podCreationTimestamp="2026-05-28 08:03:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:03:46.057216809 +0000 UTC m=+278.497289110" watchObservedRunningTime="2026-05-28 08:03:46.059120146 +0000 UTC m=+278.499192435"
May 28 08:03:46.613576 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:03:46.613536    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-q99nf" podUID="aaa6c56b-06fe-483b-9755-25d6f4930f41"
May 28 08:03:46.613736 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:03:46.613533    2551 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-cpkhd" podUID="483c9981-b25f-41e0-92e1-d8ebcf6d6966"
May 28 08:03:47.039867 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:47.036331    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 08:03:47.039867 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:47.038495    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q99nf"
May 28 08:03:50.426150 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:50.426108    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert\") pod \"ingress-canary-cpkhd\" (UID: \"483c9981-b25f-41e0-92e1-d8ebcf6d6966\") " pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 08:03:50.426749 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:50.426180    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 08:03:50.428497 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:50.428472    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa6c56b-06fe-483b-9755-25d6f4930f41-metrics-tls\") pod \"dns-default-q99nf\" (UID: \"aaa6c56b-06fe-483b-9755-25d6f4930f41\") " pod="openshift-dns/dns-default-q99nf"
May 28 08:03:50.428630 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:50.428611    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/483c9981-b25f-41e0-92e1-d8ebcf6d6966-cert\") pod \"ingress-canary-cpkhd\" (UID: \"483c9981-b25f-41e0-92e1-d8ebcf6d6966\") " pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 08:03:50.639590 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:50.639551    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-47xg9\""
May 28 08:03:50.641394 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:50.641366    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4hh68\""
May 28 08:03:50.648246 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:50.648230    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cpkhd"
May 28 08:03:50.649840 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:50.649824    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q99nf"
May 28 08:03:50.785629 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:50.785600    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q99nf"]
May 28 08:03:50.787487 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:03:50.787463    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa6c56b_06fe_483b_9755_25d6f4930f41.slice/crio-22b725576071460aa2cecb922c50dd8aa3f382162fab9f03692168096db0d1c6 WatchSource:0}: Error finding container 22b725576071460aa2cecb922c50dd8aa3f382162fab9f03692168096db0d1c6: Status 404 returned error can't find the container with id 22b725576071460aa2cecb922c50dd8aa3f382162fab9f03692168096db0d1c6
May 28 08:03:50.801977 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:50.801934    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cpkhd"]
May 28 08:03:50.803895 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:03:50.803869    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483c9981_b25f_41e0_92e1_d8ebcf6d6966.slice/crio-7fd6316369f32ed86561c68a60f836cd1eadb73646d2fe2c2521d1c6eea3af21 WatchSource:0}: Error finding container 7fd6316369f32ed86561c68a60f836cd1eadb73646d2fe2c2521d1c6eea3af21: Status 404 returned error can't find the container with id 7fd6316369f32ed86561c68a60f836cd1eadb73646d2fe2c2521d1c6eea3af21
May 28 08:03:51.050258 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:51.050221    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q99nf" event={"ID":"aaa6c56b-06fe-483b-9755-25d6f4930f41","Type":"ContainerStarted","Data":"22b725576071460aa2cecb922c50dd8aa3f382162fab9f03692168096db0d1c6"}
May 28 08:03:51.051213 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:51.051189    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cpkhd" event={"ID":"483c9981-b25f-41e0-92e1-d8ebcf6d6966","Type":"ContainerStarted","Data":"7fd6316369f32ed86561c68a60f836cd1eadb73646d2fe2c2521d1c6eea3af21"}
May 28 08:03:53.066030 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:53.065976    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q99nf" event={"ID":"aaa6c56b-06fe-483b-9755-25d6f4930f41","Type":"ContainerStarted","Data":"7e7ef39745e99e06d5f6c68933455ca3cbc7ad2b15b67c3e0da119aa57d89614"}
May 28 08:03:53.066506 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:53.066047    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q99nf" event={"ID":"aaa6c56b-06fe-483b-9755-25d6f4930f41","Type":"ContainerStarted","Data":"3b65b7e1ae0f8d2768952791610902b44bbb00b6de009b62e567e63c27fa4371"}
May 28 08:03:53.066506 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:53.066294    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q99nf"
May 28 08:03:53.067456 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:53.067435    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cpkhd" event={"ID":"483c9981-b25f-41e0-92e1-d8ebcf6d6966","Type":"ContainerStarted","Data":"992749eca40ad0475232d8d21c36626a668ec98edf5b2e8b662d940aae37c6c9"}
May 28 08:03:53.082498 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:53.082448    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-q99nf" podStartSLOduration=251.281923253 podStartE2EDuration="4m13.082434767s" podCreationTimestamp="2026-05-28 07:59:40 +0000 UTC" firstStartedPulling="2026-05-28 08:03:50.789622095 +0000 UTC m=+283.229694363" lastFinishedPulling="2026-05-28 08:03:52.590133611 +0000 UTC m=+285.030205877" observedRunningTime="2026-05-28 08:03:53.082418195 +0000 UTC m=+285.522490485" watchObservedRunningTime="2026-05-28 08:03:53.082434767 +0000 UTC m=+285.522507054"
May 28 08:03:53.098765 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:53.098726    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cpkhd" podStartSLOduration=251.312600306 podStartE2EDuration="4m13.098714108s" podCreationTimestamp="2026-05-28 07:59:40 +0000 UTC" firstStartedPulling="2026-05-28 08:03:50.807739734 +0000 UTC m=+283.247812002" lastFinishedPulling="2026-05-28 08:03:52.593853523 +0000 UTC m=+285.033925804" observedRunningTime="2026-05-28 08:03:53.096819302 +0000 UTC m=+285.536891593" watchObservedRunningTime="2026-05-28 08:03:53.098714108 +0000 UTC m=+285.538786399"
May 28 08:03:54.502846 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:54.502791    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:54.503206 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:54.502862    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:54.507622 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:54.507601    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:03:55.078818 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:03:55.078784    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:04:03.074359 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:04:03.074325    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q99nf"
May 28 08:04:08.084464 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:04:08.084435    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:04:08.087363 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:04:08.087339    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:04:08.094601 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:04:08.094577    2551 kubelet.go:1628] "Image garbage collection succeeded"
May 28 08:08:00.305583 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.305506    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b566cf6bf-nxxhf"]
May 28 08:08:00.308811 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.308794    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.331688 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.331659    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b566cf6bf-nxxhf"]
May 28 08:08:00.485284 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.485249    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlpzg\" (UniqueName: \"kubernetes.io/projected/6cb634ee-00b2-4eb2-9092-acc963210689-kube-api-access-tlpzg\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.485284 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.485288    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cb634ee-00b2-4eb2-9092-acc963210689-console-oauth-config\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.485508 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.485317    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cb634ee-00b2-4eb2-9092-acc963210689-trusted-ca-bundle\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.485508 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.485420    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cb634ee-00b2-4eb2-9092-acc963210689-console-serving-cert\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.485508 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.485465    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cb634ee-00b2-4eb2-9092-acc963210689-console-config\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.485603 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.485543    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb634ee-00b2-4eb2-9092-acc963210689-service-ca\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.485603 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.485577    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cb634ee-00b2-4eb2-9092-acc963210689-oauth-serving-cert\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.586360 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.586272    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tlpzg\" (UniqueName: \"kubernetes.io/projected/6cb634ee-00b2-4eb2-9092-acc963210689-kube-api-access-tlpzg\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.586360 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.586315    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cb634ee-00b2-4eb2-9092-acc963210689-console-oauth-config\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.586360 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.586358    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cb634ee-00b2-4eb2-9092-acc963210689-trusted-ca-bundle\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.586633 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.586409    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cb634ee-00b2-4eb2-9092-acc963210689-console-serving-cert\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.586633 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.586488    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cb634ee-00b2-4eb2-9092-acc963210689-console-config\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.586633 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.586554    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb634ee-00b2-4eb2-9092-acc963210689-service-ca\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.586633 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.586584    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cb634ee-00b2-4eb2-9092-acc963210689-oauth-serving-cert\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.587283 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.587259    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cb634ee-00b2-4eb2-9092-acc963210689-console-config\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.587397 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.587346    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb634ee-00b2-4eb2-9092-acc963210689-service-ca\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.587455 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.587373    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cb634ee-00b2-4eb2-9092-acc963210689-trusted-ca-bundle\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.587455 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.587346    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cb634ee-00b2-4eb2-9092-acc963210689-oauth-serving-cert\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.588922 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.588900    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cb634ee-00b2-4eb2-9092-acc963210689-console-serving-cert\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.589004 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.588923    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cb634ee-00b2-4eb2-9092-acc963210689-console-oauth-config\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.594250 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.594229    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlpzg\" (UniqueName: \"kubernetes.io/projected/6cb634ee-00b2-4eb2-9092-acc963210689-kube-api-access-tlpzg\") pod \"console-5b566cf6bf-nxxhf\" (UID: \"6cb634ee-00b2-4eb2-9092-acc963210689\") " pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.618260 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.618235    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:00.746055 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.746025    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b566cf6bf-nxxhf"]
May 28 08:08:00.749714 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:08:00.749684    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb634ee_00b2_4eb2_9092_acc963210689.slice/crio-1687c5a75b415770917fb62fc50945abcd918f68e2d877c01b119771772c20a5 WatchSource:0}: Error finding container 1687c5a75b415770917fb62fc50945abcd918f68e2d877c01b119771772c20a5: Status 404 returned error can't find the container with id 1687c5a75b415770917fb62fc50945abcd918f68e2d877c01b119771772c20a5
May 28 08:08:00.751500 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.751481    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 08:08:00.775890 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:00.775860    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b566cf6bf-nxxhf" event={"ID":"6cb634ee-00b2-4eb2-9092-acc963210689","Type":"ContainerStarted","Data":"1687c5a75b415770917fb62fc50945abcd918f68e2d877c01b119771772c20a5"}
May 28 08:08:01.779847 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:01.779815    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b566cf6bf-nxxhf" event={"ID":"6cb634ee-00b2-4eb2-9092-acc963210689","Type":"ContainerStarted","Data":"79a85348a378b000c7579fad9aefa72061912e6a4cd052a00fc631726360cbc3"}
May 28 08:08:01.802322 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:01.802273    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5b566cf6bf-nxxhf" podStartSLOduration=1.8022587429999999 podStartE2EDuration="1.802258743s" podCreationTimestamp="2026-05-28 08:08:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:08:01.801091509 +0000 UTC m=+534.241163808" watchObservedRunningTime="2026-05-28 08:08:01.802258743 +0000 UTC m=+534.242331033"
May 28 08:08:10.618801 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:10.618713    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:10.618801 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:10.618759    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:10.623641 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:10.623617    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:10.812741 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:10.812715    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b566cf6bf-nxxhf"
May 28 08:08:10.853919 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:10.853888    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86d69c8dfd-ldf5r"]
May 28 08:08:35.878089 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:35.878041    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-86d69c8dfd-ldf5r" podUID="8f485b61-7853-4421-8b61-e02a02152a65" containerName="console" containerID="cri-o://302eee53ac233d2635ec4a60af7323fc2cf5c7059cbd68ff6deecc545ce1205a" gracePeriod=15
May 28 08:08:36.117588 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.117568    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-86d69c8dfd-ldf5r_8f485b61-7853-4421-8b61-e02a02152a65/console/0.log"
May 28 08:08:36.117707 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.117627    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:08:36.181637 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.181554    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f485b61-7853-4421-8b61-e02a02152a65-console-serving-cert\") pod \"8f485b61-7853-4421-8b61-e02a02152a65\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") "
May 28 08:08:36.181637 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.181589    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-service-ca\") pod \"8f485b61-7853-4421-8b61-e02a02152a65\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") "
May 28 08:08:36.181637 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.181605    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-console-config\") pod \"8f485b61-7853-4421-8b61-e02a02152a65\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") "
May 28 08:08:36.181637 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.181631    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f485b61-7853-4421-8b61-e02a02152a65-console-oauth-config\") pod \"8f485b61-7853-4421-8b61-e02a02152a65\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") "
May 28 08:08:36.181952 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.181666    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-trusted-ca-bundle\") pod \"8f485b61-7853-4421-8b61-e02a02152a65\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") "
May 28 08:08:36.181952 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.181715    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmztw\" (UniqueName: \"kubernetes.io/projected/8f485b61-7853-4421-8b61-e02a02152a65-kube-api-access-wmztw\") pod \"8f485b61-7853-4421-8b61-e02a02152a65\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") "
May 28 08:08:36.181952 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.181752    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-oauth-serving-cert\") pod \"8f485b61-7853-4421-8b61-e02a02152a65\" (UID: \"8f485b61-7853-4421-8b61-e02a02152a65\") "
May 28 08:08:36.182104 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.182010    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-service-ca" (OuterVolumeSpecName: "service-ca") pod "8f485b61-7853-4421-8b61-e02a02152a65" (UID: "8f485b61-7853-4421-8b61-e02a02152a65"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 08:08:36.182104 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.182065    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-console-config" (OuterVolumeSpecName: "console-config") pod "8f485b61-7853-4421-8b61-e02a02152a65" (UID: "8f485b61-7853-4421-8b61-e02a02152a65"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 08:08:36.182303 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.182280    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8f485b61-7853-4421-8b61-e02a02152a65" (UID: "8f485b61-7853-4421-8b61-e02a02152a65"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 08:08:36.182303 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.182274    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8f485b61-7853-4421-8b61-e02a02152a65" (UID: "8f485b61-7853-4421-8b61-e02a02152a65"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 08:08:36.183848 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.183822    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f485b61-7853-4421-8b61-e02a02152a65-kube-api-access-wmztw" (OuterVolumeSpecName: "kube-api-access-wmztw") pod "8f485b61-7853-4421-8b61-e02a02152a65" (UID: "8f485b61-7853-4421-8b61-e02a02152a65"). InnerVolumeSpecName "kube-api-access-wmztw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:08:36.183954 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.183897    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f485b61-7853-4421-8b61-e02a02152a65-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8f485b61-7853-4421-8b61-e02a02152a65" (UID: "8f485b61-7853-4421-8b61-e02a02152a65"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:08:36.184006 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.183944    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f485b61-7853-4421-8b61-e02a02152a65-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8f485b61-7853-4421-8b61-e02a02152a65" (UID: "8f485b61-7853-4421-8b61-e02a02152a65"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:08:36.283435 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.283369    2551 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-service-ca\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:08:36.283435 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.283420    2551 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-console-config\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:08:36.283435 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.283435    2551 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f485b61-7853-4421-8b61-e02a02152a65-console-oauth-config\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:08:36.283667 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.283450    2551 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-trusted-ca-bundle\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:08:36.283667 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.283464    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmztw\" (UniqueName: \"kubernetes.io/projected/8f485b61-7853-4421-8b61-e02a02152a65-kube-api-access-wmztw\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:08:36.283667 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.283476    2551 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f485b61-7853-4421-8b61-e02a02152a65-oauth-serving-cert\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:08:36.283667 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.283487    2551 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f485b61-7853-4421-8b61-e02a02152a65-console-serving-cert\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:08:36.891692 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.891661    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-86d69c8dfd-ldf5r_8f485b61-7853-4421-8b61-e02a02152a65/console/0.log"
May 28 08:08:36.892177 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.891701    2551 generic.go:358] "Generic (PLEG): container finished" podID="8f485b61-7853-4421-8b61-e02a02152a65" containerID="302eee53ac233d2635ec4a60af7323fc2cf5c7059cbd68ff6deecc545ce1205a" exitCode=2
May 28 08:08:36.892177 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.891774    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86d69c8dfd-ldf5r"
May 28 08:08:36.892177 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.891805    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d69c8dfd-ldf5r" event={"ID":"8f485b61-7853-4421-8b61-e02a02152a65","Type":"ContainerDied","Data":"302eee53ac233d2635ec4a60af7323fc2cf5c7059cbd68ff6deecc545ce1205a"}
May 28 08:08:36.892177 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.891856    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86d69c8dfd-ldf5r" event={"ID":"8f485b61-7853-4421-8b61-e02a02152a65","Type":"ContainerDied","Data":"e1d5f0eced7bd1f992d3adf0ef2a4c3f570d3e63b62b084ed176b9daa69ce067"}
May 28 08:08:36.892177 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.891877    2551 scope.go:117] "RemoveContainer" containerID="302eee53ac233d2635ec4a60af7323fc2cf5c7059cbd68ff6deecc545ce1205a"
May 28 08:08:36.899914 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.899898    2551 scope.go:117] "RemoveContainer" containerID="302eee53ac233d2635ec4a60af7323fc2cf5c7059cbd68ff6deecc545ce1205a"
May 28 08:08:36.900138 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:08:36.900121    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"302eee53ac233d2635ec4a60af7323fc2cf5c7059cbd68ff6deecc545ce1205a\": container with ID starting with 302eee53ac233d2635ec4a60af7323fc2cf5c7059cbd68ff6deecc545ce1205a not found: ID does not exist" containerID="302eee53ac233d2635ec4a60af7323fc2cf5c7059cbd68ff6deecc545ce1205a"
May 28 08:08:36.900199 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.900146    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"302eee53ac233d2635ec4a60af7323fc2cf5c7059cbd68ff6deecc545ce1205a"} err="failed to get container status \"302eee53ac233d2635ec4a60af7323fc2cf5c7059cbd68ff6deecc545ce1205a\": rpc error: code = NotFound desc = could not find container \"302eee53ac233d2635ec4a60af7323fc2cf5c7059cbd68ff6deecc545ce1205a\": container with ID starting with 302eee53ac233d2635ec4a60af7323fc2cf5c7059cbd68ff6deecc545ce1205a not found: ID does not exist"
May 28 08:08:36.907950 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.907929    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86d69c8dfd-ldf5r"]
May 28 08:08:36.913333 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:36.913315    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86d69c8dfd-ldf5r"]
May 28 08:08:38.195027 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:38.194994    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f485b61-7853-4421-8b61-e02a02152a65" path="/var/lib/kubelet/pods/8f485b61-7853-4421-8b61-e02a02152a65/volumes"
May 28 08:08:45.836882 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:45.836851    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r"]
May 28 08:08:45.837235 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:45.837190    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f485b61-7853-4421-8b61-e02a02152a65" containerName="console"
May 28 08:08:45.837235 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:45.837204    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f485b61-7853-4421-8b61-e02a02152a65" containerName="console"
May 28 08:08:45.837304 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:45.837272    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="8f485b61-7853-4421-8b61-e02a02152a65" containerName="console"
May 28 08:08:45.841505 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:45.841485    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r"
May 28 08:08:45.843815 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:45.843793    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\""
May 28 08:08:45.843909 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:45.843886    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-8hv84\""
May 28 08:08:45.844932 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:45.844916    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\""
May 28 08:08:45.848113 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:45.848094    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r"]
May 28 08:08:45.970841 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:45.970805    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtlbv\" (UniqueName: \"kubernetes.io/projected/d80bd210-9c9b-478e-9c52-ab9a8317393f-kube-api-access-gtlbv\") pod \"feast-operator-controller-manager-5d8f84db5f-gvx2r\" (UID: \"d80bd210-9c9b-478e-9c52-ab9a8317393f\") " pod="feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r"
May 28 08:08:46.071468 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:46.071427    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gtlbv\" (UniqueName: \"kubernetes.io/projected/d80bd210-9c9b-478e-9c52-ab9a8317393f-kube-api-access-gtlbv\") pod \"feast-operator-controller-manager-5d8f84db5f-gvx2r\" (UID: \"d80bd210-9c9b-478e-9c52-ab9a8317393f\") " pod="feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r"
May 28 08:08:46.079259 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:46.079230    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtlbv\" (UniqueName: \"kubernetes.io/projected/d80bd210-9c9b-478e-9c52-ab9a8317393f-kube-api-access-gtlbv\") pod \"feast-operator-controller-manager-5d8f84db5f-gvx2r\" (UID: \"d80bd210-9c9b-478e-9c52-ab9a8317393f\") " pod="feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r"
May 28 08:08:46.151713 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:46.151646    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r"
May 28 08:08:46.276297 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:46.276270    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r"]
May 28 08:08:46.278867 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:08:46.278836    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80bd210_9c9b_478e_9c52_ab9a8317393f.slice/crio-9e63f5a660c665523892d267e98b010f4767cf7fa839c80b068e3bd099b081a7 WatchSource:0}: Error finding container 9e63f5a660c665523892d267e98b010f4767cf7fa839c80b068e3bd099b081a7: Status 404 returned error can't find the container with id 9e63f5a660c665523892d267e98b010f4767cf7fa839c80b068e3bd099b081a7
May 28 08:08:46.925318 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:46.925264    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r" event={"ID":"d80bd210-9c9b-478e-9c52-ab9a8317393f","Type":"ContainerStarted","Data":"9e63f5a660c665523892d267e98b010f4767cf7fa839c80b068e3bd099b081a7"}
May 28 08:08:49.935982 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:49.935945    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r" event={"ID":"d80bd210-9c9b-478e-9c52-ab9a8317393f","Type":"ContainerStarted","Data":"da333272e50fa6949222a52a7c09bdd0b0b1f45139caa8390d853953721eca50"}
May 28 08:08:49.936368 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:49.936066    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r"
May 28 08:08:49.951124 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:08:49.951077    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r" podStartSLOduration=2.07112886 podStartE2EDuration="4.951063498s" podCreationTimestamp="2026-05-28 08:08:45 +0000 UTC" firstStartedPulling="2026-05-28 08:08:46.280520274 +0000 UTC m=+578.720592546" lastFinishedPulling="2026-05-28 08:08:49.160454906 +0000 UTC m=+581.600527184" observedRunningTime="2026-05-28 08:08:49.949282494 +0000 UTC m=+582.389354805" watchObservedRunningTime="2026-05-28 08:08:49.951063498 +0000 UTC m=+582.391135784"
May 28 08:09:00.943215 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:00.943181    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r"
May 28 08:09:08.108761 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:08.108734    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:09:08.113933 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:08.113910    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:09:28.395150 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.395067    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"]
May 28 08:09:28.404916 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.404890    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.409691 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.409317    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 28 08:09:28.409691 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.409430    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 28 08:09:28.409691 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.409578    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"]
May 28 08:09:28.409691 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.409670    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 08:09:28.410049 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.409696    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 08:09:28.410049 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.409741    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-cjkt7\""
May 28 08:09:28.410049 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.409955    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 28 08:09:28.410049 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.410014    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 28 08:09:28.410262 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.410124    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 28 08:09:28.542430 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.542361    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-offline-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.542637 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.542507    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/02d69c29-d2b0-426e-aab7-3ecff721375f-feast-data\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.542637 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.542543    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-ui-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.542637 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.542582    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-online-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.542799 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.542642    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n5jf\" (UniqueName: \"kubernetes.io/projected/02d69c29-d2b0-426e-aab7-3ecff721375f-kube-api-access-7n5jf\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.542799 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.542670    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-registry-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.644030 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.643989    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-offline-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.644228 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.644042    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/02d69c29-d2b0-426e-aab7-3ecff721375f-feast-data\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.644228 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.644062    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-ui-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.644228 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.644085    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-online-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.644228 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.644116    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7n5jf\" (UniqueName: \"kubernetes.io/projected/02d69c29-d2b0-426e-aab7-3ecff721375f-kube-api-access-7n5jf\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.644228 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:09:28.644204    2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 28 08:09:28.644520 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.644252    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-registry-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.644520 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:09:28.644286    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-ui-tls podName:02d69c29-d2b0-426e-aab7-3ecff721375f nodeName:}" failed. No retries permitted until 2026-05-28 08:09:29.144264168 +0000 UTC m=+621.584336436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-ui-tls") pod "feast-simple-feast-setup-7ffbd458f8-rpgmr" (UID: "02d69c29-d2b0-426e-aab7-3ecff721375f") : secret "feast-simple-feast-setup-ui-tls" not found
May 28 08:09:28.644727 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.644708    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/02d69c29-d2b0-426e-aab7-3ecff721375f-feast-data\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.646532 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.646476    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-offline-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.646913 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.646894    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-registry-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.647008 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.646964    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-online-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:28.654088 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:28.654070    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n5jf\" (UniqueName: \"kubernetes.io/projected/02d69c29-d2b0-426e-aab7-3ecff721375f-kube-api-access-7n5jf\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:29.148676 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:29.148645    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-ui-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:29.151002 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:29.150975    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-ui-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-rpgmr\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:29.334600 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:29.334556    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:09:29.466230 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:29.466204    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"]
May 28 08:09:29.467404 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:09:29.467364    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02d69c29_d2b0_426e_aab7_3ecff721375f.slice/crio-ca977e2a81af4c0306e4b7cd3f91701a0a398867d05cdbde7d527edeb8c8e586 WatchSource:0}: Error finding container ca977e2a81af4c0306e4b7cd3f91701a0a398867d05cdbde7d527edeb8c8e586: Status 404 returned error can't find the container with id ca977e2a81af4c0306e4b7cd3f91701a0a398867d05cdbde7d527edeb8c8e586
May 28 08:09:30.055422 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:30.055369    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" event={"ID":"02d69c29-d2b0-426e-aab7-3ecff721375f","Type":"ContainerStarted","Data":"ca977e2a81af4c0306e4b7cd3f91701a0a398867d05cdbde7d527edeb8c8e586"}
May 28 08:09:32.116458 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:09:32.116405    2551 prober.go:120] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-248.ec2.internal" podUID="fc0a0cad24c1a6672f2575bc0047f126" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
May 28 08:10:08.187840 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:08.187763    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" event={"ID":"02d69c29-d2b0-426e-aab7-3ecff721375f","Type":"ContainerStarted","Data":"29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967"}
May 28 08:10:13.204546 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:13.204508    2551 generic.go:358] "Generic (PLEG): container finished" podID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerID="29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967" exitCode=0
May 28 08:10:13.204919 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:13.204580    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" event={"ID":"02d69c29-d2b0-426e-aab7-3ecff721375f","Type":"ContainerDied","Data":"29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967"}
May 28 08:10:14.209552 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:14.209514    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" event={"ID":"02d69c29-d2b0-426e-aab7-3ecff721375f","Type":"ContainerStarted","Data":"47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931"}
May 28 08:10:19.225726 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:19.225689    2551 generic.go:358] "Generic (PLEG): container finished" podID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerID="47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931" exitCode=0
May 28 08:10:19.226238 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:19.225763    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" event={"ID":"02d69c29-d2b0-426e-aab7-3ecff721375f","Type":"ContainerDied","Data":"47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931"}
May 28 08:10:19.234876 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:19.234858    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\""
May 28 08:10:20.233302 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:20.233262    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" event={"ID":"02d69c29-d2b0-426e-aab7-3ecff721375f","Type":"ContainerStarted","Data":"e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34"}
May 28 08:10:20.233806 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:20.233313    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" event={"ID":"02d69c29-d2b0-426e-aab7-3ecff721375f","Type":"ContainerStarted","Data":"732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36"}
May 28 08:10:20.233806 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:20.233328    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" event={"ID":"02d69c29-d2b0-426e-aab7-3ecff721375f","Type":"ContainerStarted","Data":"38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c"}
May 28 08:10:20.233806 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:20.233341    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" event={"ID":"02d69c29-d2b0-426e-aab7-3ecff721375f","Type":"ContainerStarted","Data":"237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66"}
May 28 08:10:20.257064 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:20.256717    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podStartSLOduration=13.899990893 podStartE2EDuration="52.256694892s" podCreationTimestamp="2026-05-28 08:09:28 +0000 UTC" firstStartedPulling="2026-05-28 08:09:29.469280812 +0000 UTC m=+621.909353079" lastFinishedPulling="2026-05-28 08:10:07.825984808 +0000 UTC m=+660.266057078" observedRunningTime="2026-05-28 08:10:20.255250921 +0000 UTC m=+672.695323256" watchObservedRunningTime="2026-05-28 08:10:20.256694892 +0000 UTC m=+672.696767182"
May 28 08:10:20.335873 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:20.335766    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:10:20.335873 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:20.335818    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:10:20.335873 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:20.335831    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:10:20.335873 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:20.335843    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:10:20.337865 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:20.337828    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused"
May 28 08:10:20.338016 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:20.337974    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 28 08:10:20.338143 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:20.337861    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.25:8816: connect: connection refused"
May 28 08:10:20.338143 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:20.337887    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.25:6571: connect: connection refused"
May 28 08:10:23.335953 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:23.335897    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused"
May 28 08:10:23.335953 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:23.335924    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 28 08:10:23.335953 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:23.335892    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.25:6571: connect: connection refused"
May 28 08:10:23.336622 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:23.335892    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.25:8816: connect: connection refused"
May 28 08:10:26.341491 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:26.341454    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:10:26.341913 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:26.341666    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:10:26.342043 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:26.341931    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:10:26.342252 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:26.342232    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:10:26.342326 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:26.342262    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:10:26.342735 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:26.342533    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:10:26.342735 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:26.342652    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:10:26.342889 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:26.342802    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:10:26.342966 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:26.342950    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:10:26.347447 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:26.347428    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:10:27.259209 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:27.259177    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:10:27.262195 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:10:27.262170    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:11:29.664805 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:29.664761    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"]
May 28 08:11:29.665398 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:29.665248    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="registry" containerID="cri-o://237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66" gracePeriod=30
May 28 08:11:29.665550 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:29.665491    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="offline" containerID="cri-o://732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36" gracePeriod=30
May 28 08:11:29.665622 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:29.665502    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="online" containerID="cri-o://38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c" gracePeriod=30
May 28 08:11:29.665724 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:29.665685    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="ui" containerID="cri-o://e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34" gracePeriod=30
May 28 08:11:29.937941 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:29.937837    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"]
May 28 08:11:29.941771 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:29.941749    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:29.944013 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:29.943992    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-pnvm6\""
May 28 08:11:29.954980 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:29.954956    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"]
May 28 08:11:30.003921 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.003872    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n9jb\" (UniqueName: \"kubernetes.io/projected/32a62eb0-2831-49eb-8ab9-1fee44fef61c-kube-api-access-2n9jb\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.003921 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.003924    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-online-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.004170 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.003988    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-registry-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.004170 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.004016    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-ui-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.004170 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.004044    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-offline-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.004170 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.004071    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/32a62eb0-2831-49eb-8ab9-1fee44fef61c-feast-data\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.104907 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.104871    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2n9jb\" (UniqueName: \"kubernetes.io/projected/32a62eb0-2831-49eb-8ab9-1fee44fef61c-kube-api-access-2n9jb\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.105021 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.104924    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-online-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.105021 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.104982    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-registry-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.105021 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.105010    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-ui-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.105192 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.105036    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-offline-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.105192 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.105062    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/32a62eb0-2831-49eb-8ab9-1fee44fef61c-feast-data\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.105294 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:11:30.105267    2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 28 08:11:30.105413 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:11:30.105289    2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 28 08:11:30.105413 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:11:30.105346    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-ui-tls podName:32a62eb0-2831-49eb-8ab9-1fee44fef61c nodeName:}" failed. No retries permitted until 2026-05-28 08:11:30.605324012 +0000 UTC m=+743.045396281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-ui-tls") pod "feast-simple-feast-setup-7ffbd458f8-hgz2k" (UID: "32a62eb0-2831-49eb-8ab9-1fee44fef61c") : secret "feast-simple-feast-setup-ui-tls" not found
May 28 08:11:30.105413 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:11:30.105372    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-registry-tls podName:32a62eb0-2831-49eb-8ab9-1fee44fef61c nodeName:}" failed. No retries permitted until 2026-05-28 08:11:30.605361878 +0000 UTC m=+743.045434153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-registry-tls") pod "feast-simple-feast-setup-7ffbd458f8-hgz2k" (UID: "32a62eb0-2831-49eb-8ab9-1fee44fef61c") : secret "feast-simple-feast-setup-registry-tls" not found
May 28 08:11:30.105777 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.105759    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/32a62eb0-2831-49eb-8ab9-1fee44fef61c-feast-data\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.107898 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.107868    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-offline-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.108088 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.108069    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-online-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.116174 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.116145    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n9jb\" (UniqueName: \"kubernetes.io/projected/32a62eb0-2831-49eb-8ab9-1fee44fef61c-kube-api-access-2n9jb\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.444400 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.444360    2551 generic.go:358] "Generic (PLEG): container finished" podID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerID="e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34" exitCode=0
May 28 08:11:30.444527 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.444436    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" event={"ID":"02d69c29-d2b0-426e-aab7-3ecff721375f","Type":"ContainerDied","Data":"e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34"}
May 28 08:11:30.610036 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.610007    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-registry-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.610036 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.610040    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-ui-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.612405 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.612355    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-registry-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.612405 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.612370    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-ui-tls\") pod \"feast-simple-feast-setup-7ffbd458f8-hgz2k\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") " pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.857232 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.857198    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:30.988942 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:30.988919    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"]
May 28 08:11:30.990529 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:11:30.990502    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32a62eb0_2831_49eb_8ab9_1fee44fef61c.slice/crio-6851a540721e4bf5be1bdc8b7f9ddc5de91945c45fcbeaee3f568cab3a9f56b4 WatchSource:0}: Error finding container 6851a540721e4bf5be1bdc8b7f9ddc5de91945c45fcbeaee3f568cab3a9f56b4: Status 404 returned error can't find the container with id 6851a540721e4bf5be1bdc8b7f9ddc5de91945c45fcbeaee3f568cab3a9f56b4
May 28 08:11:31.449277 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:31.449240    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" event={"ID":"32a62eb0-2831-49eb-8ab9-1fee44fef61c","Type":"ContainerStarted","Data":"6052894ed768931400a1fa60540473a96868f03c0ba4d2d9e24f69cad1856858"}
May 28 08:11:31.449487 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:31.449285    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" event={"ID":"32a62eb0-2831-49eb-8ab9-1fee44fef61c","Type":"ContainerStarted","Data":"6851a540721e4bf5be1bdc8b7f9ddc5de91945c45fcbeaee3f568cab3a9f56b4"}
May 28 08:11:31.451952 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:31.451929    2551 generic.go:358] "Generic (PLEG): container finished" podID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerID="38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c" exitCode=0
May 28 08:11:31.452065 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:31.451978    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" event={"ID":"02d69c29-d2b0-426e-aab7-3ecff721375f","Type":"ContainerDied","Data":"38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c"}
May 28 08:11:36.342878 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:36.342836    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused"
May 28 08:11:36.469253 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:36.469219    2551 generic.go:358] "Generic (PLEG): container finished" podID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerID="6052894ed768931400a1fa60540473a96868f03c0ba4d2d9e24f69cad1856858" exitCode=0
May 28 08:11:36.469419 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:36.469260    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" event={"ID":"32a62eb0-2831-49eb-8ab9-1fee44fef61c","Type":"ContainerDied","Data":"6052894ed768931400a1fa60540473a96868f03c0ba4d2d9e24f69cad1856858"}
May 28 08:11:37.260249 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:37.260208    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 28 08:11:37.474347 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:37.474316    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" event={"ID":"32a62eb0-2831-49eb-8ab9-1fee44fef61c","Type":"ContainerStarted","Data":"4f75ad660c462d6aa53fc61e9976d2abf29a157050e31cd40efa4e348ab04219"}
May 28 08:11:42.491082 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:42.491041    2551 generic.go:358] "Generic (PLEG): container finished" podID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerID="4f75ad660c462d6aa53fc61e9976d2abf29a157050e31cd40efa4e348ab04219" exitCode=0
May 28 08:11:42.491082 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:42.491085    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" event={"ID":"32a62eb0-2831-49eb-8ab9-1fee44fef61c","Type":"ContainerDied","Data":"4f75ad660c462d6aa53fc61e9976d2abf29a157050e31cd40efa4e348ab04219"}
May 28 08:11:43.498531 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:43.498489    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" event={"ID":"32a62eb0-2831-49eb-8ab9-1fee44fef61c","Type":"ContainerStarted","Data":"ff2d50df85e749e5eb5f572c6ebf14a8f8f393bcb5ec391edd28f48112794564"}
May 28 08:11:43.499015 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:43.498538    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" event={"ID":"32a62eb0-2831-49eb-8ab9-1fee44fef61c","Type":"ContainerStarted","Data":"df5d28dbebf1975b7b5fd5238358b23470111b051ac6cb2717a32974503661fa"}
May 28 08:11:43.499015 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:43.498555    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" event={"ID":"32a62eb0-2831-49eb-8ab9-1fee44fef61c","Type":"ContainerStarted","Data":"8257f262e2fd8b44195361b5f42da5adf3797d0f29f1905d5ebf316ee7c224c5"}
May 28 08:11:43.499015 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:43.498570    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" event={"ID":"32a62eb0-2831-49eb-8ab9-1fee44fef61c","Type":"ContainerStarted","Data":"efa497cae8d35a800634b4fb6e1d57514f8eb65e25a7ec009dfbee994570e651"}
May 28 08:11:43.522236 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:43.522175    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" podStartSLOduration=14.522154166 podStartE2EDuration="14.522154166s" podCreationTimestamp="2026-05-28 08:11:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:11:43.521006273 +0000 UTC m=+755.961078575" watchObservedRunningTime="2026-05-28 08:11:43.522154166 +0000 UTC m=+755.962226456"
May 28 08:11:45.857484 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:45.857443    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:45.858050 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:45.858022    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:45.858050 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:45.858047    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:45.858241 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:45.858060    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:45.859845 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:45.859813    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.26:6571: connect: connection refused"
May 28 08:11:45.859845 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:45.859828    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused"
May 28 08:11:45.860006 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:45.859895    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.26:8816: connect: connection refused"
May 28 08:11:45.860006 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:45.859904    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 28 08:11:46.342737 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:46.342693    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused"
May 28 08:11:47.260441 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:47.260376    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 28 08:11:48.858526 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:48.858444    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:48.858906 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:48.858710    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:48.858906 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:48.858898    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:48.859301 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:48.859224    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:48.859301 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:48.859276    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:48.859301 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:48.859290    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:48.859564 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:48.859476    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:48.859564 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:48.859554    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:48.859717 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:48.859703    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:48.863319 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:48.863301    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:49.523132 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:49.523099    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:49.526018 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:49.525989    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:11:56.343201 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:56.343149    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused"
May 28 08:11:56.343641 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:56.343299    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:11:57.260248 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:57.260196    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused"
May 28 08:11:57.260461 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:11:57.260327    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:12:00.317614 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.317592    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:12:00.480602 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.480571    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-offline-tls\") pod \"02d69c29-d2b0-426e-aab7-3ecff721375f\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") "
May 28 08:12:00.480602 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.480605    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-registry-tls\") pod \"02d69c29-d2b0-426e-aab7-3ecff721375f\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") "
May 28 08:12:00.480823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.480637    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/02d69c29-d2b0-426e-aab7-3ecff721375f-feast-data\") pod \"02d69c29-d2b0-426e-aab7-3ecff721375f\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") "
May 28 08:12:00.480823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.480688    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-ui-tls\") pod \"02d69c29-d2b0-426e-aab7-3ecff721375f\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") "
May 28 08:12:00.480823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.480725    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-online-tls\") pod \"02d69c29-d2b0-426e-aab7-3ecff721375f\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") "
May 28 08:12:00.480823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.480750    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n5jf\" (UniqueName: \"kubernetes.io/projected/02d69c29-d2b0-426e-aab7-3ecff721375f-kube-api-access-7n5jf\") pod \"02d69c29-d2b0-426e-aab7-3ecff721375f\" (UID: \"02d69c29-d2b0-426e-aab7-3ecff721375f\") "
May 28 08:12:00.481225 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.481175    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d69c29-d2b0-426e-aab7-3ecff721375f-feast-data" (OuterVolumeSpecName: "feast-data") pod "02d69c29-d2b0-426e-aab7-3ecff721375f" (UID: "02d69c29-d2b0-426e-aab7-3ecff721375f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 08:12:00.483101 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.483054    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-online-tls" (OuterVolumeSpecName: "online-tls") pod "02d69c29-d2b0-426e-aab7-3ecff721375f" (UID: "02d69c29-d2b0-426e-aab7-3ecff721375f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:12:00.483101 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.483077    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "02d69c29-d2b0-426e-aab7-3ecff721375f" (UID: "02d69c29-d2b0-426e-aab7-3ecff721375f"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:12:00.483240 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.483149    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d69c29-d2b0-426e-aab7-3ecff721375f-kube-api-access-7n5jf" (OuterVolumeSpecName: "kube-api-access-7n5jf") pod "02d69c29-d2b0-426e-aab7-3ecff721375f" (UID: "02d69c29-d2b0-426e-aab7-3ecff721375f"). InnerVolumeSpecName "kube-api-access-7n5jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:12:00.483240 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.483201    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "02d69c29-d2b0-426e-aab7-3ecff721375f" (UID: "02d69c29-d2b0-426e-aab7-3ecff721375f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:12:00.483240 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.483209    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "02d69c29-d2b0-426e-aab7-3ecff721375f" (UID: "02d69c29-d2b0-426e-aab7-3ecff721375f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:12:00.559231 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.559195    2551 generic.go:358] "Generic (PLEG): container finished" podID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerID="732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36" exitCode=137
May 28 08:12:00.559231 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.559223    2551 generic.go:358] "Generic (PLEG): container finished" podID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerID="237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66" exitCode=137
May 28 08:12:00.559498 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.559264    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"
May 28 08:12:00.559498 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.559284    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" event={"ID":"02d69c29-d2b0-426e-aab7-3ecff721375f","Type":"ContainerDied","Data":"732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36"}
May 28 08:12:00.559498 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.559329    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" event={"ID":"02d69c29-d2b0-426e-aab7-3ecff721375f","Type":"ContainerDied","Data":"237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66"}
May 28 08:12:00.559498 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.559343    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr" event={"ID":"02d69c29-d2b0-426e-aab7-3ecff721375f","Type":"ContainerDied","Data":"ca977e2a81af4c0306e4b7cd3f91701a0a398867d05cdbde7d527edeb8c8e586"}
May 28 08:12:00.559498 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.559362    2551 scope.go:117] "RemoveContainer" containerID="e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34"
May 28 08:12:00.567956 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.567939    2551 scope.go:117] "RemoveContainer" containerID="732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36"
May 28 08:12:00.575268 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.575252    2551 scope.go:117] "RemoveContainer" containerID="38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c"
May 28 08:12:00.581419 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.581396    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"]
May 28 08:12:00.581655 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.581632    2551 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-ui-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:12:00.581704 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.581664    2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-online-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:12:00.581704 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.581682    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n5jf\" (UniqueName: \"kubernetes.io/projected/02d69c29-d2b0-426e-aab7-3ecff721375f-kube-api-access-7n5jf\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:12:00.581704 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.581697    2551 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-offline-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:12:00.581806 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.581713    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/02d69c29-d2b0-426e-aab7-3ecff721375f-registry-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:12:00.581806 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.581729    2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/02d69c29-d2b0-426e-aab7-3ecff721375f-feast-data\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:12:00.583171 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.583146    2551 scope.go:117] "RemoveContainer" containerID="237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66"
May 28 08:12:00.584524 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.584504    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7ffbd458f8-rpgmr"]
May 28 08:12:00.590999 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.590983    2551 scope.go:117] "RemoveContainer" containerID="47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931"
May 28 08:12:00.597821 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.597801    2551 scope.go:117] "RemoveContainer" containerID="29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967"
May 28 08:12:00.604588 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.604571    2551 scope.go:117] "RemoveContainer" containerID="e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34"
May 28 08:12:00.604849 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:12:00.604829    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34\": container with ID starting with e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34 not found: ID does not exist" containerID="e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34"
May 28 08:12:00.604891 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.604858    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34"} err="failed to get container status \"e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34\": rpc error: code = NotFound desc = could not find container \"e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34\": container with ID starting with e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34 not found: ID does not exist"
May 28 08:12:00.604891 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.604879    2551 scope.go:117] "RemoveContainer" containerID="732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36"
May 28 08:12:00.605098 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:12:00.605082    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36\": container with ID starting with 732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36 not found: ID does not exist" containerID="732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36"
May 28 08:12:00.605135 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.605107    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36"} err="failed to get container status \"732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36\": rpc error: code = NotFound desc = could not find container \"732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36\": container with ID starting with 732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36 not found: ID does not exist"
May 28 08:12:00.605135 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.605123    2551 scope.go:117] "RemoveContainer" containerID="38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c"
May 28 08:12:00.605317 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:12:00.605302    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c\": container with ID starting with 38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c not found: ID does not exist" containerID="38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c"
May 28 08:12:00.605356 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.605320    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c"} err="failed to get container status \"38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c\": rpc error: code = NotFound desc = could not find container \"38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c\": container with ID starting with 38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c not found: ID does not exist"
May 28 08:12:00.605356 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.605332    2551 scope.go:117] "RemoveContainer" containerID="237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66"
May 28 08:12:00.605577 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:12:00.605561    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66\": container with ID starting with 237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66 not found: ID does not exist" containerID="237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66"
May 28 08:12:00.605616 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.605582    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66"} err="failed to get container status \"237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66\": rpc error: code = NotFound desc = could not find container \"237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66\": container with ID starting with 237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66 not found: ID does not exist"
May 28 08:12:00.605616 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.605596    2551 scope.go:117] "RemoveContainer" containerID="47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931"
May 28 08:12:00.605812 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:12:00.605795    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931\": container with ID starting with 47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931 not found: ID does not exist" containerID="47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931"
May 28 08:12:00.605865 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.605819    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931"} err="failed to get container status \"47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931\": rpc error: code = NotFound desc = could not find container \"47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931\": container with ID starting with 47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931 not found: ID does not exist"
May 28 08:12:00.605865 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.605841    2551 scope.go:117] "RemoveContainer" containerID="29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967"
May 28 08:12:00.606050 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:12:00.606036    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967\": container with ID starting with 29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967 not found: ID does not exist" containerID="29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967"
May 28 08:12:00.606092 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.606054    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967"} err="failed to get container status \"29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967\": rpc error: code = NotFound desc = could not find container \"29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967\": container with ID starting with 29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967 not found: ID does not exist"
May 28 08:12:00.606092 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.606066    2551 scope.go:117] "RemoveContainer" containerID="e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34"
May 28 08:12:00.606247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.606232    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34"} err="failed to get container status \"e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34\": rpc error: code = NotFound desc = could not find container \"e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34\": container with ID starting with e951c85dc0e933ec47ac7c4abcb1f91bb371a441de4f1d90dc1f562a226c4c34 not found: ID does not exist"
May 28 08:12:00.606306 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.606247    2551 scope.go:117] "RemoveContainer" containerID="732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36"
May 28 08:12:00.606549 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.606526    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36"} err="failed to get container status \"732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36\": rpc error: code = NotFound desc = could not find container \"732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36\": container with ID starting with 732a22f3ab921ddfc80dcd02e51160fc03ddc677c3bcfe804d3836607a380a36 not found: ID does not exist"
May 28 08:12:00.606602 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.606551    2551 scope.go:117] "RemoveContainer" containerID="38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c"
May 28 08:12:00.606758 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.606740    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c"} err="failed to get container status \"38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c\": rpc error: code = NotFound desc = could not find container \"38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c\": container with ID starting with 38e3a7745b7ad2e28f96e65e52ac5294f059fa5d5ea9f75f526bf61b5cb32e9c not found: ID does not exist"
May 28 08:12:00.606808 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.606758    2551 scope.go:117] "RemoveContainer" containerID="237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66"
May 28 08:12:00.606968 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.606949    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66"} err="failed to get container status \"237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66\": rpc error: code = NotFound desc = could not find container \"237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66\": container with ID starting with 237e3cd87ec471364d4c60a155d022393afbb0c53295de978f13892af2811f66 not found: ID does not exist"
May 28 08:12:00.607032 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.606970    2551 scope.go:117] "RemoveContainer" containerID="47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931"
May 28 08:12:00.607164 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.607148    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931"} err="failed to get container status \"47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931\": rpc error: code = NotFound desc = could not find container \"47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931\": container with ID starting with 47762ee4769953931838b2b7862f33aea92f0a4340b35ea1445c3b884009b931 not found: ID does not exist"
May 28 08:12:00.607230 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.607166    2551 scope.go:117] "RemoveContainer" containerID="29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967"
May 28 08:12:00.607405 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:00.607361    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967"} err="failed to get container status \"29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967\": rpc error: code = NotFound desc = could not find container \"29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967\": container with ID starting with 29458aeb2b00599ab2d58447bf9676ef7ba64bc809611218cd1e017c56deb967 not found: ID does not exist"
May 28 08:12:02.194646 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:02.194615    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" path="/var/lib/kubelet/pods/02d69c29-d2b0-426e-aab7-3ecff721375f/volumes"
May 28 08:12:51.505897 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.505851    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"]
May 28 08:12:51.506405 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506326    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="ui"
May 28 08:12:51.506405 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506340    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="ui"
May 28 08:12:51.506405 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506355    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="registry"
May 28 08:12:51.506405 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506366    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="registry"
May 28 08:12:51.506405 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506396    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="online"
May 28 08:12:51.506405 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506405    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="online"
May 28 08:12:51.506607 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506429    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="offline"
May 28 08:12:51.506607 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506438    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="offline"
May 28 08:12:51.506607 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506457    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="feast-init"
May 28 08:12:51.506607 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506464    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="feast-init"
May 28 08:12:51.506607 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506475    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="feast-apply"
May 28 08:12:51.506607 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506482    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="feast-apply"
May 28 08:12:51.506607 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506555    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="online"
May 28 08:12:51.506607 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506567    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="offline"
May 28 08:12:51.506607 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506578    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="ui"
May 28 08:12:51.506607 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.506588    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="02d69c29-d2b0-426e-aab7-3ecff721375f" containerName="registry"
May 28 08:12:51.510577 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.510558    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.513075 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.513052    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 28 08:12:51.513237 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.513218    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 28 08:12:51.513321 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.513279    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-q2rks\""
May 28 08:12:51.513400 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.513318    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 28 08:12:51.513400 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.513179    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 28 08:12:51.513518 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.513120    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 28 08:12:51.513518 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.513116    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 28 08:12:51.513518 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.513224    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\""
May 28 08:12:51.521073 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.521051    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"]
May 28 08:12:51.614569 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.614533    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8jh\" (UniqueName: \"kubernetes.io/projected/16d6bf51-7036-4a51-b403-bb97891aa1f4-kube-api-access-qn8jh\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.614707 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.614627    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/16d6bf51-7036-4a51-b403-bb97891aa1f4-feast-data\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.614707 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.614659    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-online-tls\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.614707 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.614693    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-offline-tls\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.614807 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.614721    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-ui-tls\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.614807 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.614747    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/16d6bf51-7036-4a51-b403-bb97891aa1f4-registry-tls\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.715947 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.715915    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-offline-tls\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.716117 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.715955    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-ui-tls\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.716117 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.715984    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/16d6bf51-7036-4a51-b403-bb97891aa1f4-registry-tls\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.716117 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.716031    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8jh\" (UniqueName: \"kubernetes.io/projected/16d6bf51-7036-4a51-b403-bb97891aa1f4-kube-api-access-qn8jh\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.716117 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:12:51.716057    2551 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found
May 28 08:12:51.716358 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.716117    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/16d6bf51-7036-4a51-b403-bb97891aa1f4-feast-data\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.716358 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:12:51.716134    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-offline-tls podName:16d6bf51-7036-4a51-b403-bb97891aa1f4 nodeName:}" failed. No retries permitted until 2026-05-28 08:12:52.216112824 +0000 UTC m=+824.656185105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-offline-tls") pod "feast-simple-feast-remote-setup-b9c649c8f-mjq64" (UID: "16d6bf51-7036-4a51-b403-bb97891aa1f4") : secret "feast-simple-feast-remote-setup-offline-tls" not found
May 28 08:12:51.716358 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.716178    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-online-tls\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.716358 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:12:51.716295    2551 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found
May 28 08:12:51.716358 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:12:51.716336    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-online-tls podName:16d6bf51-7036-4a51-b403-bb97891aa1f4 nodeName:}" failed. No retries permitted until 2026-05-28 08:12:52.216322586 +0000 UTC m=+824.656394870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-online-tls") pod "feast-simple-feast-remote-setup-b9c649c8f-mjq64" (UID: "16d6bf51-7036-4a51-b403-bb97891aa1f4") : secret "feast-simple-feast-remote-setup-online-tls" not found
May 28 08:12:51.716904 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.716878    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/16d6bf51-7036-4a51-b403-bb97891aa1f4-registry-tls\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.717006 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.716936    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/16d6bf51-7036-4a51-b403-bb97891aa1f4-feast-data\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.718967 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.718943    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-ui-tls\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:51.724446 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:51.724423    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8jh\" (UniqueName: \"kubernetes.io/projected/16d6bf51-7036-4a51-b403-bb97891aa1f4-kube-api-access-qn8jh\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:52.220956 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:52.220918    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-online-tls\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:52.220956 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:52.220962    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-offline-tls\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:52.223457 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:52.223427    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-online-tls\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:52.223570 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:52.223487    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-offline-tls\") pod \"feast-simple-feast-remote-setup-b9c649c8f-mjq64\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:52.422949 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:52.422911    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:12:52.554498 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:52.554469    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"]
May 28 08:12:52.554855 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:12:52.554831    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16d6bf51_7036_4a51_b403_bb97891aa1f4.slice/crio-b19930ca0e5114db914dbb9a704b28a291ef0c8d2a531ff399f3a0883efd8091 WatchSource:0}: Error finding container b19930ca0e5114db914dbb9a704b28a291ef0c8d2a531ff399f3a0883efd8091: Status 404 returned error can't find the container with id b19930ca0e5114db914dbb9a704b28a291ef0c8d2a531ff399f3a0883efd8091
May 28 08:12:52.722884 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:52.722843    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" event={"ID":"16d6bf51-7036-4a51-b403-bb97891aa1f4","Type":"ContainerStarted","Data":"fadcc1c96bc0a85b521bd553209bb6aac92d48d4a9cbb9706af61e221e218db3"}
May 28 08:12:52.722884 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:52.722886    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" event={"ID":"16d6bf51-7036-4a51-b403-bb97891aa1f4","Type":"ContainerStarted","Data":"b19930ca0e5114db914dbb9a704b28a291ef0c8d2a531ff399f3a0883efd8091"}
May 28 08:12:57.739278 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:57.739240    2551 generic.go:358] "Generic (PLEG): container finished" podID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerID="fadcc1c96bc0a85b521bd553209bb6aac92d48d4a9cbb9706af61e221e218db3" exitCode=0
May 28 08:12:57.739725 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:57.739311    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" event={"ID":"16d6bf51-7036-4a51-b403-bb97891aa1f4","Type":"ContainerDied","Data":"fadcc1c96bc0a85b521bd553209bb6aac92d48d4a9cbb9706af61e221e218db3"}
May 28 08:12:58.744842 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:12:58.744809    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" event={"ID":"16d6bf51-7036-4a51-b403-bb97891aa1f4","Type":"ContainerStarted","Data":"1cd865882571ec465a939e0fbead9628aab8929de1459835fe9e85c37e1d2d2a"}
May 28 08:13:03.761057 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:03.761022    2551 generic.go:358] "Generic (PLEG): container finished" podID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerID="1cd865882571ec465a939e0fbead9628aab8929de1459835fe9e85c37e1d2d2a" exitCode=0
May 28 08:13:03.761474 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:03.761094    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" event={"ID":"16d6bf51-7036-4a51-b403-bb97891aa1f4","Type":"ContainerDied","Data":"1cd865882571ec465a939e0fbead9628aab8929de1459835fe9e85c37e1d2d2a"}
May 28 08:13:03.762320 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:03.762303    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 08:13:04.767964 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:04.767928    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" event={"ID":"16d6bf51-7036-4a51-b403-bb97891aa1f4","Type":"ContainerStarted","Data":"f22d67a8e76b310e5c2204cf0bdcedf9dec0c153d3d74ac3949356ac18009747"}
May 28 08:13:04.767964 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:04.767969    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" event={"ID":"16d6bf51-7036-4a51-b403-bb97891aa1f4","Type":"ContainerStarted","Data":"301707bf28dfe73489fcd54f46c0637968b9f896ecbf62d6e3b0eef3013f313d"}
May 28 08:13:04.768541 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:04.767980    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" event={"ID":"16d6bf51-7036-4a51-b403-bb97891aa1f4","Type":"ContainerStarted","Data":"5a2c5a710923e67b3d71f5b45c060a4c5792c0f7e62b1de75c55ede21867bcf5"}
May 28 08:13:04.793815 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:04.793748    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" podStartSLOduration=13.793727986 podStartE2EDuration="13.793727986s" podCreationTimestamp="2026-05-28 08:12:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:13:04.791118399 +0000 UTC m=+837.231190713" watchObservedRunningTime="2026-05-28 08:13:04.793727986 +0000 UTC m=+837.233800276"
May 28 08:13:07.423957 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:07.423915    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:13:07.424480 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:07.423984    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:13:07.424480 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:07.423999    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:13:07.425977 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:07.425944    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.27:8816: connect: connection refused"
May 28 08:13:07.425977 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:07.425962    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused"
May 28 08:13:07.425977 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:07.425972    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused"
May 28 08:13:10.424149 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:10.424116    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:13:10.424586 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:10.424278    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:13:10.424718 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:10.424697    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:13:10.424806 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:10.424730    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:13:10.424954 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:10.424933    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:13:10.425065 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:10.425052    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:13:10.428252 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:10.428233    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:13:10.791336 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:10.791301    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:13:10.794348 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:13:10.794329    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:14:08.135956 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:08.135887    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:14:08.138272 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:08.138251    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:14:12.756602 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:12.756567    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"]
May 28 08:14:12.757059 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:12.756969    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="online" containerID="cri-o://5a2c5a710923e67b3d71f5b45c060a4c5792c0f7e62b1de75c55ede21867bcf5" gracePeriod=30
May 28 08:14:12.757191 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:12.757169    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="ui" containerID="cri-o://f22d67a8e76b310e5c2204cf0bdcedf9dec0c153d3d74ac3949356ac18009747" gracePeriod=30
May 28 08:14:12.757335 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:12.757303    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="offline" containerID="cri-o://301707bf28dfe73489fcd54f46c0637968b9f896ecbf62d6e3b0eef3013f313d" gracePeriod=30
May 28 08:14:12.867036 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:12.867001    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"]
May 28 08:14:12.867516 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:12.867478    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="registry" containerID="cri-o://efa497cae8d35a800634b4fb6e1d57514f8eb65e25a7ec009dfbee994570e651" gracePeriod=30
May 28 08:14:12.867687 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:12.867505    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="ui" containerID="cri-o://ff2d50df85e749e5eb5f572c6ebf14a8f8f393bcb5ec391edd28f48112794564" gracePeriod=30
May 28 08:14:12.867687 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:12.867627    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="online" containerID="cri-o://8257f262e2fd8b44195361b5f42da5adf3797d0f29f1905d5ebf316ee7c224c5" gracePeriod=30
May 28 08:14:12.867687 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:12.867609    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="offline" containerID="cri-o://df5d28dbebf1975b7b5fd5238358b23470111b051ac6cb2717a32974503661fa" gracePeriod=30
May 28 08:14:13.982914 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:13.982884    2551 generic.go:358] "Generic (PLEG): container finished" podID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerID="ff2d50df85e749e5eb5f572c6ebf14a8f8f393bcb5ec391edd28f48112794564" exitCode=0
May 28 08:14:13.982914 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:13.982908    2551 generic.go:358] "Generic (PLEG): container finished" podID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerID="8257f262e2fd8b44195361b5f42da5adf3797d0f29f1905d5ebf316ee7c224c5" exitCode=0
May 28 08:14:13.983410 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:13.982952    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" event={"ID":"32a62eb0-2831-49eb-8ab9-1fee44fef61c","Type":"ContainerDied","Data":"ff2d50df85e749e5eb5f572c6ebf14a8f8f393bcb5ec391edd28f48112794564"}
May 28 08:14:13.983410 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:13.982984    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" event={"ID":"32a62eb0-2831-49eb-8ab9-1fee44fef61c","Type":"ContainerDied","Data":"8257f262e2fd8b44195361b5f42da5adf3797d0f29f1905d5ebf316ee7c224c5"}
May 28 08:14:13.985176 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:13.985156    2551 generic.go:358] "Generic (PLEG): container finished" podID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerID="f22d67a8e76b310e5c2204cf0bdcedf9dec0c153d3d74ac3949356ac18009747" exitCode=0
May 28 08:14:13.985176 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:13.985173    2551 generic.go:358] "Generic (PLEG): container finished" podID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerID="5a2c5a710923e67b3d71f5b45c060a4c5792c0f7e62b1de75c55ede21867bcf5" exitCode=0
May 28 08:14:13.985298 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:13.985191    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" event={"ID":"16d6bf51-7036-4a51-b403-bb97891aa1f4","Type":"ContainerDied","Data":"f22d67a8e76b310e5c2204cf0bdcedf9dec0c153d3d74ac3949356ac18009747"}
May 28 08:14:13.985298 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:13.985209    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" event={"ID":"16d6bf51-7036-4a51-b403-bb97891aa1f4","Type":"ContainerDied","Data":"5a2c5a710923e67b3d71f5b45c060a4c5792c0f7e62b1de75c55ede21867bcf5"}
May 28 08:14:18.860177 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:18.860125    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused"
May 28 08:14:19.523982 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:19.523939    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 28 08:14:20.425576 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:20.425528    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused"
May 28 08:14:20.792012 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:20.791973    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused"
May 28 08:14:28.859822 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:28.859772    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused"
May 28 08:14:29.523787 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:29.523746    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 28 08:14:30.425199 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:30.425159    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused"
May 28 08:14:30.791823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:30.791778    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused"
May 28 08:14:38.859895 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:38.859813    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused"
May 28 08:14:38.860319 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:38.860044    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:14:39.523565 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:39.523525    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused"
May 28 08:14:39.523732 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:39.523635    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:14:40.425521 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:40.425469    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused"
May 28 08:14:40.425934 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:40.425600    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:14:40.792245 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:40.792203    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused"
May 28 08:14:40.792459 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:40.792317    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:14:43.074120 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.074034    2551 generic.go:358] "Generic (PLEG): container finished" podID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerID="301707bf28dfe73489fcd54f46c0637968b9f896ecbf62d6e3b0eef3013f313d" exitCode=137
May 28 08:14:43.074536 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.074114    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" event={"ID":"16d6bf51-7036-4a51-b403-bb97891aa1f4","Type":"ContainerDied","Data":"301707bf28dfe73489fcd54f46c0637968b9f896ecbf62d6e3b0eef3013f313d"}
May 28 08:14:43.076603 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.076578    2551 generic.go:358] "Generic (PLEG): container finished" podID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerID="df5d28dbebf1975b7b5fd5238358b23470111b051ac6cb2717a32974503661fa" exitCode=137
May 28 08:14:43.076603 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.076598    2551 generic.go:358] "Generic (PLEG): container finished" podID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerID="efa497cae8d35a800634b4fb6e1d57514f8eb65e25a7ec009dfbee994570e651" exitCode=137
May 28 08:14:43.076768 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.076639    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" event={"ID":"32a62eb0-2831-49eb-8ab9-1fee44fef61c","Type":"ContainerDied","Data":"df5d28dbebf1975b7b5fd5238358b23470111b051ac6cb2717a32974503661fa"}
May 28 08:14:43.076768 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.076674    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" event={"ID":"32a62eb0-2831-49eb-8ab9-1fee44fef61c","Type":"ContainerDied","Data":"efa497cae8d35a800634b4fb6e1d57514f8eb65e25a7ec009dfbee994570e651"}
May 28 08:14:43.410955 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.410917    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:14:43.528747 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.528720    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:14:43.571807 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.571775    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/16d6bf51-7036-4a51-b403-bb97891aa1f4-feast-data\") pod \"16d6bf51-7036-4a51-b403-bb97891aa1f4\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") "
May 28 08:14:43.571972 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.571816    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn8jh\" (UniqueName: \"kubernetes.io/projected/16d6bf51-7036-4a51-b403-bb97891aa1f4-kube-api-access-qn8jh\") pod \"16d6bf51-7036-4a51-b403-bb97891aa1f4\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") "
May 28 08:14:43.571972 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.571863    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-offline-tls\") pod \"16d6bf51-7036-4a51-b403-bb97891aa1f4\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") "
May 28 08:14:43.571972 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.571899    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-ui-tls\") pod \"16d6bf51-7036-4a51-b403-bb97891aa1f4\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") "
May 28 08:14:43.571972 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.571928    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/16d6bf51-7036-4a51-b403-bb97891aa1f4-registry-tls\") pod \"16d6bf51-7036-4a51-b403-bb97891aa1f4\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") "
May 28 08:14:43.572181 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.572064    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-online-tls\") pod \"16d6bf51-7036-4a51-b403-bb97891aa1f4\" (UID: \"16d6bf51-7036-4a51-b403-bb97891aa1f4\") "
May 28 08:14:43.572329 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.572293    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d6bf51-7036-4a51-b403-bb97891aa1f4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "16d6bf51-7036-4a51-b403-bb97891aa1f4" (UID: "16d6bf51-7036-4a51-b403-bb97891aa1f4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 08:14:43.572416 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.572359    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d6bf51-7036-4a51-b403-bb97891aa1f4-feast-data" (OuterVolumeSpecName: "feast-data") pod "16d6bf51-7036-4a51-b403-bb97891aa1f4" (UID: "16d6bf51-7036-4a51-b403-bb97891aa1f4"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 08:14:43.572476 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.572413    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/16d6bf51-7036-4a51-b403-bb97891aa1f4-registry-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:14:43.574206 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.574180    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-online-tls" (OuterVolumeSpecName: "online-tls") pod "16d6bf51-7036-4a51-b403-bb97891aa1f4" (UID: "16d6bf51-7036-4a51-b403-bb97891aa1f4"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:14:43.574206 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.574192    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d6bf51-7036-4a51-b403-bb97891aa1f4-kube-api-access-qn8jh" (OuterVolumeSpecName: "kube-api-access-qn8jh") pod "16d6bf51-7036-4a51-b403-bb97891aa1f4" (UID: "16d6bf51-7036-4a51-b403-bb97891aa1f4"). InnerVolumeSpecName "kube-api-access-qn8jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:14:43.574334 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.574207    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "16d6bf51-7036-4a51-b403-bb97891aa1f4" (UID: "16d6bf51-7036-4a51-b403-bb97891aa1f4"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:14:43.574334 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.574312    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "16d6bf51-7036-4a51-b403-bb97891aa1f4" (UID: "16d6bf51-7036-4a51-b403-bb97891aa1f4"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:14:43.673213 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.673129    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-ui-tls\") pod \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") "
May 28 08:14:43.673213 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.673177    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n9jb\" (UniqueName: \"kubernetes.io/projected/32a62eb0-2831-49eb-8ab9-1fee44fef61c-kube-api-access-2n9jb\") pod \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") "
May 28 08:14:43.673213 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.673203    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-registry-tls\") pod \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") "
May 28 08:14:43.673507 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.673232    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-offline-tls\") pod \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") "
May 28 08:14:43.673507 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.673264    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-online-tls\") pod \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") "
May 28 08:14:43.673507 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.673284    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/32a62eb0-2831-49eb-8ab9-1fee44fef61c-feast-data\") pod \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\" (UID: \"32a62eb0-2831-49eb-8ab9-1fee44fef61c\") "
May 28 08:14:43.673507 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.673490    2551 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-offline-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:14:43.673507 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.673507    2551 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-ui-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:14:43.673744 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.673521    2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/16d6bf51-7036-4a51-b403-bb97891aa1f4-online-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:14:43.673744 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.673536    2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/16d6bf51-7036-4a51-b403-bb97891aa1f4-feast-data\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:14:43.673744 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.673550    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn8jh\" (UniqueName: \"kubernetes.io/projected/16d6bf51-7036-4a51-b403-bb97891aa1f4-kube-api-access-qn8jh\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:14:43.674006 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.673979    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a62eb0-2831-49eb-8ab9-1fee44fef61c-feast-data" (OuterVolumeSpecName: "feast-data") pod "32a62eb0-2831-49eb-8ab9-1fee44fef61c" (UID: "32a62eb0-2831-49eb-8ab9-1fee44fef61c"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 08:14:43.675482 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.675460    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "32a62eb0-2831-49eb-8ab9-1fee44fef61c" (UID: "32a62eb0-2831-49eb-8ab9-1fee44fef61c"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:14:43.675826 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.675798    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-online-tls" (OuterVolumeSpecName: "online-tls") pod "32a62eb0-2831-49eb-8ab9-1fee44fef61c" (UID: "32a62eb0-2831-49eb-8ab9-1fee44fef61c"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:14:43.675826 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.675815    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "32a62eb0-2831-49eb-8ab9-1fee44fef61c" (UID: "32a62eb0-2831-49eb-8ab9-1fee44fef61c"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:14:43.675952 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.675845    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a62eb0-2831-49eb-8ab9-1fee44fef61c-kube-api-access-2n9jb" (OuterVolumeSpecName: "kube-api-access-2n9jb") pod "32a62eb0-2831-49eb-8ab9-1fee44fef61c" (UID: "32a62eb0-2831-49eb-8ab9-1fee44fef61c"). InnerVolumeSpecName "kube-api-access-2n9jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:14:43.675952 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.675863    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "32a62eb0-2831-49eb-8ab9-1fee44fef61c" (UID: "32a62eb0-2831-49eb-8ab9-1fee44fef61c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:14:43.774393 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.774354    2551 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-ui-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:14:43.774509 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.774393    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n9jb\" (UniqueName: \"kubernetes.io/projected/32a62eb0-2831-49eb-8ab9-1fee44fef61c-kube-api-access-2n9jb\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:14:43.774509 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.774415    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-registry-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:14:43.774509 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.774424    2551 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-offline-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:14:43.774509 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.774434    2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/32a62eb0-2831-49eb-8ab9-1fee44fef61c-online-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:14:43.774509 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:43.774442    2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/32a62eb0-2831-49eb-8ab9-1fee44fef61c-feast-data\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:14:44.081439 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.081403    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64" event={"ID":"16d6bf51-7036-4a51-b403-bb97891aa1f4","Type":"ContainerDied","Data":"b19930ca0e5114db914dbb9a704b28a291ef0c8d2a531ff399f3a0883efd8091"}
May 28 08:14:44.081975 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.081448    2551 scope.go:117] "RemoveContainer" containerID="f22d67a8e76b310e5c2204cf0bdcedf9dec0c153d3d74ac3949356ac18009747"
May 28 08:14:44.081975 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.081414    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"
May 28 08:14:44.084173 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.084152    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k" event={"ID":"32a62eb0-2831-49eb-8ab9-1fee44fef61c","Type":"ContainerDied","Data":"6851a540721e4bf5be1bdc8b7f9ddc5de91945c45fcbeaee3f568cab3a9f56b4"}
May 28 08:14:44.084298 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.084213    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"
May 28 08:14:44.090983 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.090957    2551 scope.go:117] "RemoveContainer" containerID="301707bf28dfe73489fcd54f46c0637968b9f896ecbf62d6e3b0eef3013f313d"
May 28 08:14:44.098309 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.098290    2551 scope.go:117] "RemoveContainer" containerID="5a2c5a710923e67b3d71f5b45c060a4c5792c0f7e62b1de75c55ede21867bcf5"
May 28 08:14:44.103813 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.103791    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"]
May 28 08:14:44.107061 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.107043    2551 scope.go:117] "RemoveContainer" containerID="1cd865882571ec465a939e0fbead9628aab8929de1459835fe9e85c37e1d2d2a"
May 28 08:14:44.107827 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.107804    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b9c649c8f-mjq64"]
May 28 08:14:44.113887 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.113864    2551 scope.go:117] "RemoveContainer" containerID="fadcc1c96bc0a85b521bd553209bb6aac92d48d4a9cbb9706af61e221e218db3"
May 28 08:14:44.118338 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.118314    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"]
May 28 08:14:44.121684 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.121660    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-7ffbd458f8-hgz2k"]
May 28 08:14:44.121899 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.121884    2551 scope.go:117] "RemoveContainer" containerID="ff2d50df85e749e5eb5f572c6ebf14a8f8f393bcb5ec391edd28f48112794564"
May 28 08:14:44.129045 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.129029    2551 scope.go:117] "RemoveContainer" containerID="df5d28dbebf1975b7b5fd5238358b23470111b051ac6cb2717a32974503661fa"
May 28 08:14:44.135559 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.135544    2551 scope.go:117] "RemoveContainer" containerID="8257f262e2fd8b44195361b5f42da5adf3797d0f29f1905d5ebf316ee7c224c5"
May 28 08:14:44.142395 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.142363    2551 scope.go:117] "RemoveContainer" containerID="efa497cae8d35a800634b4fb6e1d57514f8eb65e25a7ec009dfbee994570e651"
May 28 08:14:44.148713 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.148696    2551 scope.go:117] "RemoveContainer" containerID="4f75ad660c462d6aa53fc61e9976d2abf29a157050e31cd40efa4e348ab04219"
May 28 08:14:44.155203 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.155188    2551 scope.go:117] "RemoveContainer" containerID="6052894ed768931400a1fa60540473a96868f03c0ba4d2d9e24f69cad1856858"
May 28 08:14:44.194977 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.194947    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" path="/var/lib/kubelet/pods/16d6bf51-7036-4a51-b403-bb97891aa1f4/volumes"
May 28 08:14:44.200365 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:14:44.196011    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" path="/var/lib/kubelet/pods/32a62eb0-2831-49eb-8ab9-1fee44fef61c/volumes"
May 28 08:15:06.444856 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.444815    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"]
May 28 08:15:06.446103 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.446077    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="feast-apply"
May 28 08:15:06.446237 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.446226    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="feast-apply"
May 28 08:15:06.446327 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.446318    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="registry"
May 28 08:15:06.446419 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.446410    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="registry"
May 28 08:15:06.446509 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.446500    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="feast-apply"
May 28 08:15:06.446582 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.446574    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="feast-apply"
May 28 08:15:06.446653 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.446645    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="offline"
May 28 08:15:06.446717 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.446709    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="offline"
May 28 08:15:06.446794 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.446785    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="feast-init"
May 28 08:15:06.446857 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.446849    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="feast-init"
May 28 08:15:06.446935 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.446927    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="offline"
May 28 08:15:06.446996 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.446989    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="offline"
May 28 08:15:06.447079 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.447071    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="online"
May 28 08:15:06.447143 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.447135    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="online"
May 28 08:15:06.447218 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.447210    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="ui"
May 28 08:15:06.447280 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.447273    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="ui"
May 28 08:15:06.447344 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.447336    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="online"
May 28 08:15:06.447431 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.447422    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="online"
May 28 08:15:06.447526 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.447517    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="feast-init"
May 28 08:15:06.447595 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.447584    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="feast-init"
May 28 08:15:06.447660 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.447651    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="ui"
May 28 08:15:06.447724 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.447716    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="ui"
May 28 08:15:06.447937 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.447924    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="registry"
May 28 08:15:06.448018 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.448010    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="ui"
May 28 08:15:06.448096 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.448088    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="offline"
May 28 08:15:06.448157 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.448149    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="offline"
May 28 08:15:06.448225 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.448217    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="ui"
May 28 08:15:06.448285 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.448277    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="32a62eb0-2831-49eb-8ab9-1fee44fef61c" containerName="online"
May 28 08:15:06.448352 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.448344    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="16d6bf51-7036-4a51-b403-bb97891aa1f4" containerName="online"
May 28 08:15:06.453660 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.453633    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:06.456152 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.456129    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\""
May 28 08:15:06.456292 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.456270    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\""
May 28 08:15:06.456363 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.456321    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\""
May 28 08:15:06.456363 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.456341    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\""
May 28 08:15:06.456525 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.456164    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 08:15:06.457084 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.457057    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-qlnc5\""
May 28 08:15:06.457223 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.457104    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 28 08:15:06.457342 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.457318    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 08:15:06.460537 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.460513    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"]
May 28 08:15:06.559777 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.559732    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-registry-tls\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:06.559959 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.559790    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/719272cb-4094-4a1e-88d2-57f48009e2a4-feast-data\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:06.559959 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.559820    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n7d4\" (UniqueName: \"kubernetes.io/projected/719272cb-4094-4a1e-88d2-57f48009e2a4-kube-api-access-9n7d4\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:06.559959 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.559850    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-online-tls\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:06.559959 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.559893    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-offline-tls\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:06.660582 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.660546    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-registry-tls\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:06.660582 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.660584    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/719272cb-4094-4a1e-88d2-57f48009e2a4-feast-data\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:06.660799 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.660611    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9n7d4\" (UniqueName: \"kubernetes.io/projected/719272cb-4094-4a1e-88d2-57f48009e2a4-kube-api-access-9n7d4\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:06.660799 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.660644    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-online-tls\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:06.660799 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.660685    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-offline-tls\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:06.660799 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:15:06.660699    2551 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found
May 28 08:15:06.660799 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:15:06.660780    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-registry-tls podName:719272cb-4094-4a1e-88d2-57f48009e2a4 nodeName:}" failed. No retries permitted until 2026-05-28 08:15:07.160757735 +0000 UTC m=+959.600830019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-registry-tls") pod "feast-credit-scoring-767ddb7866-qddnz" (UID: "719272cb-4094-4a1e-88d2-57f48009e2a4") : secret "feast-credit-scoring-registry-tls" not found
May 28 08:15:06.661035 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.661010    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/719272cb-4094-4a1e-88d2-57f48009e2a4-feast-data\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:06.663071 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.663044    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-offline-tls\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:06.663210 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.663117    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-online-tls\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:06.669035 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:06.669006    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n7d4\" (UniqueName: \"kubernetes.io/projected/719272cb-4094-4a1e-88d2-57f48009e2a4-kube-api-access-9n7d4\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:07.165368 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:07.165333    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-registry-tls\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:07.167750 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:07.167725    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-registry-tls\") pod \"feast-credit-scoring-767ddb7866-qddnz\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") " pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:07.365423 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:07.365370    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:07.495582 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:07.495554    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"]
May 28 08:15:07.497006 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:15:07.496979    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719272cb_4094_4a1e_88d2_57f48009e2a4.slice/crio-78270db7308e3a2b703abbbe89582fe5f429574dee55a957848dbb2bd8ebd50d WatchSource:0}: Error finding container 78270db7308e3a2b703abbbe89582fe5f429574dee55a957848dbb2bd8ebd50d: Status 404 returned error can't find the container with id 78270db7308e3a2b703abbbe89582fe5f429574dee55a957848dbb2bd8ebd50d
May 28 08:15:08.156921 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:08.156879    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" event={"ID":"719272cb-4094-4a1e-88d2-57f48009e2a4","Type":"ContainerStarted","Data":"7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753"}
May 28 08:15:08.156921 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:08.156924    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" event={"ID":"719272cb-4094-4a1e-88d2-57f48009e2a4","Type":"ContainerStarted","Data":"78270db7308e3a2b703abbbe89582fe5f429574dee55a957848dbb2bd8ebd50d"}
May 28 08:15:09.161534 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:09.161498    2551 generic.go:358] "Generic (PLEG): container finished" podID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerID="7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753" exitCode=0
May 28 08:15:09.162005 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:09.161553    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" event={"ID":"719272cb-4094-4a1e-88d2-57f48009e2a4","Type":"ContainerDied","Data":"7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753"}
May 28 08:15:09.164869 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:09.164849    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\""
May 28 08:15:10.169461 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:10.169428    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" event={"ID":"719272cb-4094-4a1e-88d2-57f48009e2a4","Type":"ContainerStarted","Data":"6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844"}
May 28 08:15:16.189449 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:16.189415    2551 generic.go:358] "Generic (PLEG): container finished" podID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerID="6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844" exitCode=0
May 28 08:15:16.189845 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:16.189486    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" event={"ID":"719272cb-4094-4a1e-88d2-57f48009e2a4","Type":"ContainerDied","Data":"6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844"}
May 28 08:15:16.192796 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:16.192775    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\""
May 28 08:15:17.196075 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:17.196035    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" event={"ID":"719272cb-4094-4a1e-88d2-57f48009e2a4","Type":"ContainerStarted","Data":"77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75"}
May 28 08:15:17.196075 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:17.196078    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" event={"ID":"719272cb-4094-4a1e-88d2-57f48009e2a4","Type":"ContainerStarted","Data":"2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce"}
May 28 08:15:17.196669 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:17.196088    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" event={"ID":"719272cb-4094-4a1e-88d2-57f48009e2a4","Type":"ContainerStarted","Data":"5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5"}
May 28 08:15:17.219640 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:17.219581    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" podStartSLOduration=11.219562814 podStartE2EDuration="11.219562814s" podCreationTimestamp="2026-05-28 08:15:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:15:17.217037864 +0000 UTC m=+969.657110153" watchObservedRunningTime="2026-05-28 08:15:17.219562814 +0000 UTC m=+969.659635104"
May 28 08:15:17.365790 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:17.365748    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:17.365953 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:17.365807    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:17.365953 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:17.365823    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:19.366050 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:19.365964    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:19.366050 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:19.366015    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:19.366050 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:19.366029    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:19.368073 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:19.368037    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.28:8816: connect: connection refused"
May 28 08:15:19.368172 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:19.368074    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused"
May 28 08:15:19.368172 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:19.368046    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.28:6571: connect: connection refused"
May 28 08:15:22.365899 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:22.365847    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.28:6571: connect: connection refused"
May 28 08:15:22.366489 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:22.366460    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:22.367562 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:22.367541    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:22.371341 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:22.371320    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:23.219118 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:23.219087    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:25.366039 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:25.365999    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.28:6571: connect: connection refused"
May 28 08:15:28.366249 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:28.366219    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:15:28.366974 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:15:28.366957    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:16:31.775877 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:16:31.775836    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"]
May 28 08:16:31.776370 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:16:31.776260    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="registry" containerID="cri-o://5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5" gracePeriod=30
May 28 08:16:31.776370 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:16:31.776274    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="online" containerID="cri-o://2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce" gracePeriod=30
May 28 08:16:31.776370 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:16:31.776280    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="offline" containerID="cri-o://77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75" gracePeriod=30
May 28 08:16:33.215658 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:16:33.215618    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused"
May 28 08:16:33.431197 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:16:33.431164    2551 generic.go:358] "Generic (PLEG): container finished" podID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerID="2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce" exitCode=0
May 28 08:16:33.431360 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:16:33.431227    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" event={"ID":"719272cb-4094-4a1e-88d2-57f48009e2a4","Type":"ContainerDied","Data":"2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce"}
May 28 08:16:43.215757 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:16:43.215710    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused"
May 28 08:16:53.215288 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:16:53.215192    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused"
May 28 08:16:53.215690 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:16:53.215318    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:17:02.430074 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.430050    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:17:02.472587 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.472551    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-online-tls\") pod \"719272cb-4094-4a1e-88d2-57f48009e2a4\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") "
May 28 08:17:02.472733 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.472619    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n7d4\" (UniqueName: \"kubernetes.io/projected/719272cb-4094-4a1e-88d2-57f48009e2a4-kube-api-access-9n7d4\") pod \"719272cb-4094-4a1e-88d2-57f48009e2a4\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") "
May 28 08:17:02.472733 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.472655    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-offline-tls\") pod \"719272cb-4094-4a1e-88d2-57f48009e2a4\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") "
May 28 08:17:02.472733 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.472698    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/719272cb-4094-4a1e-88d2-57f48009e2a4-feast-data\") pod \"719272cb-4094-4a1e-88d2-57f48009e2a4\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") "
May 28 08:17:02.472895 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.472744    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-registry-tls\") pod \"719272cb-4094-4a1e-88d2-57f48009e2a4\" (UID: \"719272cb-4094-4a1e-88d2-57f48009e2a4\") "
May 28 08:17:02.474873 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.474836    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "719272cb-4094-4a1e-88d2-57f48009e2a4" (UID: "719272cb-4094-4a1e-88d2-57f48009e2a4"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:17:02.474999 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.474893    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-online-tls" (OuterVolumeSpecName: "online-tls") pod "719272cb-4094-4a1e-88d2-57f48009e2a4" (UID: "719272cb-4094-4a1e-88d2-57f48009e2a4"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:17:02.475118 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.475099    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "719272cb-4094-4a1e-88d2-57f48009e2a4" (UID: "719272cb-4094-4a1e-88d2-57f48009e2a4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:17:02.475456 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.475430    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719272cb-4094-4a1e-88d2-57f48009e2a4-kube-api-access-9n7d4" (OuterVolumeSpecName: "kube-api-access-9n7d4") pod "719272cb-4094-4a1e-88d2-57f48009e2a4" (UID: "719272cb-4094-4a1e-88d2-57f48009e2a4"). InnerVolumeSpecName "kube-api-access-9n7d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:17:02.482355 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.482333    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719272cb-4094-4a1e-88d2-57f48009e2a4-feast-data" (OuterVolumeSpecName: "feast-data") pod "719272cb-4094-4a1e-88d2-57f48009e2a4" (UID: "719272cb-4094-4a1e-88d2-57f48009e2a4"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 08:17:02.518315 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.518285    2551 generic.go:358] "Generic (PLEG): container finished" podID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerID="77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75" exitCode=137
May 28 08:17:02.518315 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.518312    2551 generic.go:358] "Generic (PLEG): container finished" podID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerID="5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5" exitCode=137
May 28 08:17:02.518471 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.518357    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"
May 28 08:17:02.518471 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.518369    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" event={"ID":"719272cb-4094-4a1e-88d2-57f48009e2a4","Type":"ContainerDied","Data":"77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75"}
May 28 08:17:02.518471 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.518432    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" event={"ID":"719272cb-4094-4a1e-88d2-57f48009e2a4","Type":"ContainerDied","Data":"5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5"}
May 28 08:17:02.518471 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.518443    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" event={"ID":"719272cb-4094-4a1e-88d2-57f48009e2a4","Type":"ContainerDied","Data":"78270db7308e3a2b703abbbe89582fe5f429574dee55a957848dbb2bd8ebd50d"}
May 28 08:17:02.518471 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.518458    2551 scope.go:117] "RemoveContainer" containerID="77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75"
May 28 08:17:02.526365 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.526341    2551 scope.go:117] "RemoveContainer" containerID="2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce"
May 28 08:17:02.534268 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.534250    2551 scope.go:117] "RemoveContainer" containerID="5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5"
May 28 08:17:02.539237 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.539213    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"]
May 28 08:17:02.542302 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.542286    2551 scope.go:117] "RemoveContainer" containerID="6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844"
May 28 08:17:02.543170 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.543156    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-767ddb7866-qddnz"]
May 28 08:17:02.549032 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.549014    2551 scope.go:117] "RemoveContainer" containerID="7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753"
May 28 08:17:02.555408 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.555377    2551 scope.go:117] "RemoveContainer" containerID="77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75"
May 28 08:17:02.555647 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:17:02.555632    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75\": container with ID starting with 77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75 not found: ID does not exist" containerID="77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75"
May 28 08:17:02.555692 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.555653    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75"} err="failed to get container status \"77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75\": rpc error: code = NotFound desc = could not find container \"77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75\": container with ID starting with 77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75 not found: ID does not exist"
May 28 08:17:02.555692 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.555669    2551 scope.go:117] "RemoveContainer" containerID="2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce"
May 28 08:17:02.555876 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:17:02.555861    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce\": container with ID starting with 2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce not found: ID does not exist" containerID="2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce"
May 28 08:17:02.555921 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.555877    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce"} err="failed to get container status \"2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce\": rpc error: code = NotFound desc = could not find container \"2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce\": container with ID starting with 2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce not found: ID does not exist"
May 28 08:17:02.555921 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.555888    2551 scope.go:117] "RemoveContainer" containerID="5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5"
May 28 08:17:02.556096 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:17:02.556078    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5\": container with ID starting with 5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5 not found: ID does not exist" containerID="5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5"
May 28 08:17:02.556154 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.556103    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5"} err="failed to get container status \"5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5\": rpc error: code = NotFound desc = could not find container \"5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5\": container with ID starting with 5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5 not found: ID does not exist"
May 28 08:17:02.556154 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.556120    2551 scope.go:117] "RemoveContainer" containerID="6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844"
May 28 08:17:02.556318 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:17:02.556300    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844\": container with ID starting with 6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844 not found: ID does not exist" containerID="6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844"
May 28 08:17:02.556361 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.556322    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844"} err="failed to get container status \"6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844\": rpc error: code = NotFound desc = could not find container \"6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844\": container with ID starting with 6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844 not found: ID does not exist"
May 28 08:17:02.556361 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.556334    2551 scope.go:117] "RemoveContainer" containerID="7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753"
May 28 08:17:02.556592 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:17:02.556572    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753\": container with ID starting with 7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753 not found: ID does not exist" containerID="7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753"
May 28 08:17:02.556649 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.556595    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753"} err="failed to get container status \"7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753\": rpc error: code = NotFound desc = could not find container \"7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753\": container with ID starting with 7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753 not found: ID does not exist"
May 28 08:17:02.556649 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.556611    2551 scope.go:117] "RemoveContainer" containerID="77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75"
May 28 08:17:02.556846 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.556824    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75"} err="failed to get container status \"77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75\": rpc error: code = NotFound desc = could not find container \"77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75\": container with ID starting with 77307addca497a55cf0edc4e40f938ef0fd62960476a5c9c19a27ba2eee08b75 not found: ID does not exist"
May 28 08:17:02.556918 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.556848    2551 scope.go:117] "RemoveContainer" containerID="2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce"
May 28 08:17:02.557072 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.557050    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce"} err="failed to get container status \"2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce\": rpc error: code = NotFound desc = could not find container \"2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce\": container with ID starting with 2a9e6fc0c8c760e3d4933aac3be83123c45d2450c8e61d2eeab1bb4aea9322ce not found: ID does not exist"
May 28 08:17:02.557141 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.557074    2551 scope.go:117] "RemoveContainer" containerID="5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5"
May 28 08:17:02.557267 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.557250    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5"} err="failed to get container status \"5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5\": rpc error: code = NotFound desc = could not find container \"5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5\": container with ID starting with 5cbfbd83bb621763ec343bda2cd140cf6c50798dfb30cac8b7ca54dcfe8c65a5 not found: ID does not exist"
May 28 08:17:02.557330 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.557268    2551 scope.go:117] "RemoveContainer" containerID="6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844"
May 28 08:17:02.557474 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.557452    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844"} err="failed to get container status \"6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844\": rpc error: code = NotFound desc = could not find container \"6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844\": container with ID starting with 6cd51b6c0ee8f7ce6fab647dd27abf87ad01b4aeea0da695bcbb24eaf5075844 not found: ID does not exist"
May 28 08:17:02.557525 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.557475    2551 scope.go:117] "RemoveContainer" containerID="7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753"
May 28 08:17:02.557668 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.557649    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753"} err="failed to get container status \"7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753\": rpc error: code = NotFound desc = could not find container \"7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753\": container with ID starting with 7a7cce2c0b78dba7090ceb1e53baeb05d5c246a6bf548fb0fff7d90db1127753 not found: ID does not exist"
May 28 08:17:02.573613 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.573580    2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-online-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:17:02.573613 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.573614    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n7d4\" (UniqueName: \"kubernetes.io/projected/719272cb-4094-4a1e-88d2-57f48009e2a4-kube-api-access-9n7d4\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:17:02.573722 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.573624    2551 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-offline-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:17:02.573722 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.573633    2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/719272cb-4094-4a1e-88d2-57f48009e2a4-feast-data\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:17:02.573722 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:02.573641    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/719272cb-4094-4a1e-88d2-57f48009e2a4-registry-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:17:03.368626 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:03.368585    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-767ddb7866-qddnz" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.28:8816: i/o timeout"
May 28 08:17:04.199871 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:17:04.199831    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" path="/var/lib/kubelet/pods/719272cb-4094-4a1e-88d2-57f48009e2a4/volumes"
May 28 08:19:08.158892 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:08.158866    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:19:08.159366 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:08.158865    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:19:12.934433 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.934393    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"]
May 28 08:19:12.934970 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.934828    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="feast-apply"
May 28 08:19:12.934970 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.934843    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="feast-apply"
May 28 08:19:12.934970 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.934866    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="registry"
May 28 08:19:12.934970 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.934874    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="registry"
May 28 08:19:12.934970 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.934885    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="online"
May 28 08:19:12.934970 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.934894    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="online"
May 28 08:19:12.934970 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.934922    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="offline"
May 28 08:19:12.934970 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.934931    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="offline"
May 28 08:19:12.934970 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.934942    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="feast-init"
May 28 08:19:12.934970 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.934951    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="feast-init"
May 28 08:19:12.935269 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.935013    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="online"
May 28 08:19:12.935269 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.935025    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="offline"
May 28 08:19:12.935269 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.935039    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="719272cb-4094-4a1e-88d2-57f48009e2a4" containerName="registry"
May 28 08:19:12.938644 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.938626    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:12.941170 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.941147    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\""
May 28 08:19:12.941398 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.941155    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\""
May 28 08:19:12.941485 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.941223    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\""
May 28 08:19:12.941544 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.941491    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\""
May 28 08:19:12.941544 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.941267    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\""
May 28 08:19:12.941644 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.941260    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\""
May 28 08:19:12.941644 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.941277    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-z5dgx\""
May 28 08:19:12.947803 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:12.947781    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"]
May 28 08:19:13.069009 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.068971    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxxjr\" (UniqueName: \"kubernetes.io/projected/de23df0f-3ab4-498f-82bb-c021b844928c-kube-api-access-pxxjr\") pod \"feast-credit-scoring-5b9745568c-9hg59\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") " pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:13.069009 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.069011    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/de23df0f-3ab4-498f-82bb-c021b844928c-registry-tls\") pod \"feast-credit-scoring-5b9745568c-9hg59\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") " pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:13.069213 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.069028    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/de23df0f-3ab4-498f-82bb-c021b844928c-online-tls\") pod \"feast-credit-scoring-5b9745568c-9hg59\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") " pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:13.069213 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.069112    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/de23df0f-3ab4-498f-82bb-c021b844928c-feast-data\") pod \"feast-credit-scoring-5b9745568c-9hg59\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") " pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:13.170679 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.170644    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pxxjr\" (UniqueName: \"kubernetes.io/projected/de23df0f-3ab4-498f-82bb-c021b844928c-kube-api-access-pxxjr\") pod \"feast-credit-scoring-5b9745568c-9hg59\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") " pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:13.170679 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.170682    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/de23df0f-3ab4-498f-82bb-c021b844928c-registry-tls\") pod \"feast-credit-scoring-5b9745568c-9hg59\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") " pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:13.170882 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.170703    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/de23df0f-3ab4-498f-82bb-c021b844928c-online-tls\") pod \"feast-credit-scoring-5b9745568c-9hg59\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") " pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:13.170882 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.170740    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/de23df0f-3ab4-498f-82bb-c021b844928c-feast-data\") pod \"feast-credit-scoring-5b9745568c-9hg59\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") " pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:13.171213 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.171177    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/de23df0f-3ab4-498f-82bb-c021b844928c-feast-data\") pod \"feast-credit-scoring-5b9745568c-9hg59\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") " pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:13.173059 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.173027    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/de23df0f-3ab4-498f-82bb-c021b844928c-online-tls\") pod \"feast-credit-scoring-5b9745568c-9hg59\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") " pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:13.173155 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.173071    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/de23df0f-3ab4-498f-82bb-c021b844928c-registry-tls\") pod \"feast-credit-scoring-5b9745568c-9hg59\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") " pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:13.177755 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.177735    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxxjr\" (UniqueName: \"kubernetes.io/projected/de23df0f-3ab4-498f-82bb-c021b844928c-kube-api-access-pxxjr\") pod \"feast-credit-scoring-5b9745568c-9hg59\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") " pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:13.248579 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.248547    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:13.378443 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.378362    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"]
May 28 08:19:13.381570 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:19:13.381529    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde23df0f_3ab4_498f_82bb_c021b844928c.slice/crio-6e96d0120b2052286ad067f862cf3b9bb071c427dc81e21964d9623c5d07eb36 WatchSource:0}: Error finding container 6e96d0120b2052286ad067f862cf3b9bb071c427dc81e21964d9623c5d07eb36: Status 404 returned error can't find the container with id 6e96d0120b2052286ad067f862cf3b9bb071c427dc81e21964d9623c5d07eb36
May 28 08:19:13.383418 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.383376    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 08:19:13.901689 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.901646    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" event={"ID":"de23df0f-3ab4-498f-82bb-c021b844928c","Type":"ContainerStarted","Data":"95c8260385612c79b43db47b39a57eb4da0b4ad3334c6a54dfba532034c6eee8"}
May 28 08:19:13.901689 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:13.901694    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" event={"ID":"de23df0f-3ab4-498f-82bb-c021b844928c","Type":"ContainerStarted","Data":"6e96d0120b2052286ad067f862cf3b9bb071c427dc81e21964d9623c5d07eb36"}
May 28 08:19:14.905821 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:14.905780    2551 generic.go:358] "Generic (PLEG): container finished" podID="de23df0f-3ab4-498f-82bb-c021b844928c" containerID="95c8260385612c79b43db47b39a57eb4da0b4ad3334c6a54dfba532034c6eee8" exitCode=0
May 28 08:19:14.906191 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:14.905872    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" event={"ID":"de23df0f-3ab4-498f-82bb-c021b844928c","Type":"ContainerDied","Data":"95c8260385612c79b43db47b39a57eb4da0b4ad3334c6a54dfba532034c6eee8"}
May 28 08:19:15.911326 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:15.911282    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" event={"ID":"de23df0f-3ab4-498f-82bb-c021b844928c","Type":"ContainerStarted","Data":"1a4bb543c33577e7b4befa0e78ea4cf362b403490b1b15e5e1a8710dd7dd74b8"}
May 28 08:19:22.934024 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:22.933988    2551 generic.go:358] "Generic (PLEG): container finished" podID="de23df0f-3ab4-498f-82bb-c021b844928c" containerID="1a4bb543c33577e7b4befa0e78ea4cf362b403490b1b15e5e1a8710dd7dd74b8" exitCode=0
May 28 08:19:22.934404 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:22.934052    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" event={"ID":"de23df0f-3ab4-498f-82bb-c021b844928c","Type":"ContainerDied","Data":"1a4bb543c33577e7b4befa0e78ea4cf362b403490b1b15e5e1a8710dd7dd74b8"}
May 28 08:19:23.939956 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:23.939920    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" event={"ID":"de23df0f-3ab4-498f-82bb-c021b844928c","Type":"ContainerStarted","Data":"78ed9d07892654416fe6a74fd433dba5257d62153de503859ba25838bb4d606a"}
May 28 08:19:23.939956 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:23.939964    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" event={"ID":"de23df0f-3ab4-498f-82bb-c021b844928c","Type":"ContainerStarted","Data":"e52f68f74c93b293fca8c157f1a1aea1c52b5362325c15b2260c7c65a60d7f91"}
May 28 08:19:23.960571 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:23.960518    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" podStartSLOduration=11.960502449 podStartE2EDuration="11.960502449s" podCreationTimestamp="2026-05-28 08:19:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:19:23.958256629 +0000 UTC m=+1216.398328930" watchObservedRunningTime="2026-05-28 08:19:23.960502449 +0000 UTC m=+1216.400574758"
May 28 08:19:25.249231 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:25.249180    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:25.249623 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:25.249242    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:25.250830 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:25.250800    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused"
May 28 08:19:25.250944 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:25.250846    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused"
May 28 08:19:28.249928 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:28.249883    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused"
May 28 08:19:28.250306 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:28.249992    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused"
May 28 08:19:31.249949 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:31.249895    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused"
May 28 08:19:31.255141 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:31.255117    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:31.255743 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:31.255721    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:31.258519 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:31.258499    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:33.249131 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:33.249096    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:34.250147 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:34.250116    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:34.977220 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:34.977187    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:19:38.796233 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.792012    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"]
May 28 08:19:38.799777 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.799751    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:38.803255 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.802821    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\""
May 28 08:19:38.803255 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.803075    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-qhnbm\""
May 28 08:19:38.803255 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.803228    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\""
May 28 08:19:38.803255 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.803398    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\""
May 28 08:19:38.805940 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.805912    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"]
May 28 08:19:38.891195 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.891163    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47b9dd47-358a-4148-a233-d56e3462f334-online-tls\") pod \"feast-driver-ranking-7f654c97b-g4ftc\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") " pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:38.891352 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.891218    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/47b9dd47-358a-4148-a233-d56e3462f334-registry-tls\") pod \"feast-driver-ranking-7f654c97b-g4ftc\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") " pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:38.891352 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.891298    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76vnm\" (UniqueName: \"kubernetes.io/projected/47b9dd47-358a-4148-a233-d56e3462f334-kube-api-access-76vnm\") pod \"feast-driver-ranking-7f654c97b-g4ftc\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") " pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:38.891352 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.891328    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/47b9dd47-358a-4148-a233-d56e3462f334-feast-data\") pod \"feast-driver-ranking-7f654c97b-g4ftc\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") " pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:38.992473 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.992425    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47b9dd47-358a-4148-a233-d56e3462f334-online-tls\") pod \"feast-driver-ranking-7f654c97b-g4ftc\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") " pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:38.992654 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.992488    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/47b9dd47-358a-4148-a233-d56e3462f334-registry-tls\") pod \"feast-driver-ranking-7f654c97b-g4ftc\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") " pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:38.992654 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.992533    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-76vnm\" (UniqueName: \"kubernetes.io/projected/47b9dd47-358a-4148-a233-d56e3462f334-kube-api-access-76vnm\") pod \"feast-driver-ranking-7f654c97b-g4ftc\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") " pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:38.992654 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.992551    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/47b9dd47-358a-4148-a233-d56e3462f334-feast-data\") pod \"feast-driver-ranking-7f654c97b-g4ftc\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") " pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:38.992654 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:19:38.992591    2551 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found
May 28 08:19:38.992881 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:19:38.992667    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47b9dd47-358a-4148-a233-d56e3462f334-online-tls podName:47b9dd47-358a-4148-a233-d56e3462f334 nodeName:}" failed. No retries permitted until 2026-05-28 08:19:39.492646141 +0000 UTC m=+1231.932718422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/47b9dd47-358a-4148-a233-d56e3462f334-online-tls") pod "feast-driver-ranking-7f654c97b-g4ftc" (UID: "47b9dd47-358a-4148-a233-d56e3462f334") : secret "feast-driver-ranking-online-tls" not found
May 28 08:19:38.992946 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.992912    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/47b9dd47-358a-4148-a233-d56e3462f334-feast-data\") pod \"feast-driver-ranking-7f654c97b-g4ftc\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") " pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:38.994975 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:38.994956    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/47b9dd47-358a-4148-a233-d56e3462f334-registry-tls\") pod \"feast-driver-ranking-7f654c97b-g4ftc\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") " pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:39.000355 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:39.000335    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76vnm\" (UniqueName: \"kubernetes.io/projected/47b9dd47-358a-4148-a233-d56e3462f334-kube-api-access-76vnm\") pod \"feast-driver-ranking-7f654c97b-g4ftc\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") " pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:39.497996 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:39.497964    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47b9dd47-358a-4148-a233-d56e3462f334-online-tls\") pod \"feast-driver-ranking-7f654c97b-g4ftc\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") " pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:39.500486 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:39.500455    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47b9dd47-358a-4148-a233-d56e3462f334-online-tls\") pod \"feast-driver-ranking-7f654c97b-g4ftc\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") " pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:39.713005 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:39.712963    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:39.837969 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:39.837934    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"]
May 28 08:19:39.841168 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:19:39.841142    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b9dd47_358a_4148_a233_d56e3462f334.slice/crio-a77cf54981c05ccea8331e0e1bbc93a0390d7512ed158d282b24254d79ca1c2b WatchSource:0}: Error finding container a77cf54981c05ccea8331e0e1bbc93a0390d7512ed158d282b24254d79ca1c2b: Status 404 returned error can't find the container with id a77cf54981c05ccea8331e0e1bbc93a0390d7512ed158d282b24254d79ca1c2b
May 28 08:19:39.993049 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:39.993010    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" event={"ID":"47b9dd47-358a-4148-a233-d56e3462f334","Type":"ContainerStarted","Data":"e96eedbceed0f7c66a8aea20c84d057b4539725f4b81bf3dd7a384a199632bbc"}
May 28 08:19:39.993049 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:39.993049    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" event={"ID":"47b9dd47-358a-4148-a233-d56e3462f334","Type":"ContainerStarted","Data":"a77cf54981c05ccea8331e0e1bbc93a0390d7512ed158d282b24254d79ca1c2b"}
May 28 08:19:45.016654 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:45.016609    2551 generic.go:358] "Generic (PLEG): container finished" podID="47b9dd47-358a-4148-a233-d56e3462f334" containerID="e96eedbceed0f7c66a8aea20c84d057b4539725f4b81bf3dd7a384a199632bbc" exitCode=0
May 28 08:19:45.017114 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:45.016685    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" event={"ID":"47b9dd47-358a-4148-a233-d56e3462f334","Type":"ContainerDied","Data":"e96eedbceed0f7c66a8aea20c84d057b4539725f4b81bf3dd7a384a199632bbc"}
May 28 08:19:46.021642 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:46.021606    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" event={"ID":"47b9dd47-358a-4148-a233-d56e3462f334","Type":"ContainerStarted","Data":"a15f108d1d89945afa9de7a8caaaa86e85be783175565d445b99f37d122ee72e"}
May 28 08:19:53.043718 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:53.043682    2551 generic.go:358] "Generic (PLEG): container finished" podID="47b9dd47-358a-4148-a233-d56e3462f334" containerID="a15f108d1d89945afa9de7a8caaaa86e85be783175565d445b99f37d122ee72e" exitCode=0
May 28 08:19:53.044077 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:53.043753    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" event={"ID":"47b9dd47-358a-4148-a233-d56e3462f334","Type":"ContainerDied","Data":"a15f108d1d89945afa9de7a8caaaa86e85be783175565d445b99f37d122ee72e"}
May 28 08:19:54.048957 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:54.048921    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" event={"ID":"47b9dd47-358a-4148-a233-d56e3462f334","Type":"ContainerStarted","Data":"95d93a1974e6c6ae1bafa4e6b5b4ed25235623df117f9e68f9c00c0d1761747f"}
May 28 08:19:54.048957 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:54.048960    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" event={"ID":"47b9dd47-358a-4148-a233-d56e3462f334","Type":"ContainerStarted","Data":"7d9bb30e5829dc66f02a19d36a025911817b50b3229917ea187fe6017306291a"}
May 28 08:19:54.070200 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:54.070132    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" podStartSLOduration=16.070112899 podStartE2EDuration="16.070112899s" podCreationTimestamp="2026-05-28 08:19:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:19:54.067236682 +0000 UTC m=+1246.507308994" watchObservedRunningTime="2026-05-28 08:19:54.070112899 +0000 UTC m=+1246.510185189"
May 28 08:19:54.713835 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:54.713798    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:54.714014 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:54.713852    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:54.715249 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:54.715221    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused"
May 28 08:19:54.715359 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:54.715237    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused"
May 28 08:19:57.713644 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:57.713600    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused"
May 28 08:19:57.714149 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:57.713724    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused"
May 28 08:19:59.713720 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:59.713689    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:19:59.714092 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:19:59.713757    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:20:00.714277 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:20:00.714220    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused"
May 28 08:20:00.719353 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:20:00.719319    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:20:01.075796 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:20:01.075709    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:20:03.713473 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:20:03.713412    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.30:6571: connect: connection refused"
May 28 08:20:06.714077 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:20:06.714044    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:20:06.714757 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:20:06.714738    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:21:23.518038 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:23.517961    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r"]
May 28 08:21:23.518536 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:23.518204    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r" podUID="d80bd210-9c9b-478e-9c52-ab9a8317393f" containerName="manager" containerID="cri-o://da333272e50fa6949222a52a7c09bdd0b0b1f45139caa8390d853953721eca50" gracePeriod=10
May 28 08:21:23.745803 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:23.745702    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"]
May 28 08:21:23.746200 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:23.746138    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="registry" containerID="cri-o://e52f68f74c93b293fca8c157f1a1aea1c52b5362325c15b2260c7c65a60d7f91" gracePeriod=30
May 28 08:21:23.746334 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:23.746235    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="online" containerID="cri-o://78ed9d07892654416fe6a74fd433dba5257d62153de503859ba25838bb4d606a" gracePeriod=30
May 28 08:21:23.758449 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:23.758418    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"]
May 28 08:21:23.758843 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:23.758793    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="registry" containerID="cri-o://7d9bb30e5829dc66f02a19d36a025911817b50b3229917ea187fe6017306291a" gracePeriod=30
May 28 08:21:23.758951 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:23.758846    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="online" containerID="cri-o://95d93a1974e6c6ae1bafa4e6b5b4ed25235623df117f9e68f9c00c0d1761747f" gracePeriod=30
May 28 08:21:23.813069 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:23.813041    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r"
May 28 08:21:23.910920 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:23.910881    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtlbv\" (UniqueName: \"kubernetes.io/projected/d80bd210-9c9b-478e-9c52-ab9a8317393f-kube-api-access-gtlbv\") pod \"d80bd210-9c9b-478e-9c52-ab9a8317393f\" (UID: \"d80bd210-9c9b-478e-9c52-ab9a8317393f\") "
May 28 08:21:23.913080 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:23.913047    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d80bd210-9c9b-478e-9c52-ab9a8317393f-kube-api-access-gtlbv" (OuterVolumeSpecName: "kube-api-access-gtlbv") pod "d80bd210-9c9b-478e-9c52-ab9a8317393f" (UID: "d80bd210-9c9b-478e-9c52-ab9a8317393f"). InnerVolumeSpecName "kube-api-access-gtlbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:21:24.012165 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:24.012127    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtlbv\" (UniqueName: \"kubernetes.io/projected/d80bd210-9c9b-478e-9c52-ab9a8317393f-kube-api-access-gtlbv\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:21:24.326931 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:24.326890    2551 generic.go:358] "Generic (PLEG): container finished" podID="d80bd210-9c9b-478e-9c52-ab9a8317393f" containerID="da333272e50fa6949222a52a7c09bdd0b0b1f45139caa8390d853953721eca50" exitCode=0
May 28 08:21:24.327117 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:24.326983    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r"
May 28 08:21:24.327117 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:24.327016    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r" event={"ID":"d80bd210-9c9b-478e-9c52-ab9a8317393f","Type":"ContainerDied","Data":"da333272e50fa6949222a52a7c09bdd0b0b1f45139caa8390d853953721eca50"}
May 28 08:21:24.327117 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:24.327058    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r" event={"ID":"d80bd210-9c9b-478e-9c52-ab9a8317393f","Type":"ContainerDied","Data":"9e63f5a660c665523892d267e98b010f4767cf7fa839c80b068e3bd099b081a7"}
May 28 08:21:24.327117 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:24.327073    2551 scope.go:117] "RemoveContainer" containerID="da333272e50fa6949222a52a7c09bdd0b0b1f45139caa8390d853953721eca50"
May 28 08:21:24.335644 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:24.335621    2551 scope.go:117] "RemoveContainer" containerID="da333272e50fa6949222a52a7c09bdd0b0b1f45139caa8390d853953721eca50"
May 28 08:21:24.335944 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:21:24.335923    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da333272e50fa6949222a52a7c09bdd0b0b1f45139caa8390d853953721eca50\": container with ID starting with da333272e50fa6949222a52a7c09bdd0b0b1f45139caa8390d853953721eca50 not found: ID does not exist" containerID="da333272e50fa6949222a52a7c09bdd0b0b1f45139caa8390d853953721eca50"
May 28 08:21:24.335998 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:24.335953    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da333272e50fa6949222a52a7c09bdd0b0b1f45139caa8390d853953721eca50"} err="failed to get container status \"da333272e50fa6949222a52a7c09bdd0b0b1f45139caa8390d853953721eca50\": rpc error: code = NotFound desc = could not find container \"da333272e50fa6949222a52a7c09bdd0b0b1f45139caa8390d853953721eca50\": container with ID starting with da333272e50fa6949222a52a7c09bdd0b0b1f45139caa8390d853953721eca50 not found: ID does not exist"
May 28 08:21:24.342753 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:24.342726    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r"]
May 28 08:21:24.345910 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:24.345881    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-5d8f84db5f-gvx2r"]
May 28 08:21:25.331937 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:25.331904    2551 generic.go:358] "Generic (PLEG): container finished" podID="47b9dd47-358a-4148-a233-d56e3462f334" containerID="95d93a1974e6c6ae1bafa4e6b5b4ed25235623df117f9e68f9c00c0d1761747f" exitCode=0
May 28 08:21:25.332378 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:25.331975    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" event={"ID":"47b9dd47-358a-4148-a233-d56e3462f334","Type":"ContainerDied","Data":"95d93a1974e6c6ae1bafa4e6b5b4ed25235623df117f9e68f9c00c0d1761747f"}
May 28 08:21:25.334677 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:25.334654    2551 generic.go:358] "Generic (PLEG): container finished" podID="de23df0f-3ab4-498f-82bb-c021b844928c" containerID="78ed9d07892654416fe6a74fd433dba5257d62153de503859ba25838bb4d606a" exitCode=0
May 28 08:21:25.334779 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:25.334699    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" event={"ID":"de23df0f-3ab4-498f-82bb-c021b844928c","Type":"ContainerDied","Data":"78ed9d07892654416fe6a74fd433dba5257d62153de503859ba25838bb4d606a"}
May 28 08:21:26.194915 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:26.194884    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d80bd210-9c9b-478e-9c52-ab9a8317393f" path="/var/lib/kubelet/pods/d80bd210-9c9b-478e-9c52-ab9a8317393f/volumes"
May 28 08:21:31.073318 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:31.073272    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused"
May 28 08:21:31.256336 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:31.256293    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused"
May 28 08:21:41.072736 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:41.072697    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused"
May 28 08:21:41.256658 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:41.256617    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused"
May 28 08:21:51.072678 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:51.072634    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="online" probeResult="failure" output="Get \"https://10.133.0.30:6567/health\": dial tcp 10.133.0.30:6567: connect: connection refused"
May 28 08:21:51.073141 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:51.072747    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:21:51.256059 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:51.256019    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused"
May 28 08:21:51.256235 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:51.256129    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:21:54.418626 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.418591    2551 generic.go:358] "Generic (PLEG): container finished" podID="de23df0f-3ab4-498f-82bb-c021b844928c" containerID="e52f68f74c93b293fca8c157f1a1aea1c52b5362325c15b2260c7c65a60d7f91" exitCode=137
May 28 08:21:54.418959 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.418649    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" event={"ID":"de23df0f-3ab4-498f-82bb-c021b844928c","Type":"ContainerDied","Data":"e52f68f74c93b293fca8c157f1a1aea1c52b5362325c15b2260c7c65a60d7f91"}
May 28 08:21:54.418959 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.418688    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59" event={"ID":"de23df0f-3ab4-498f-82bb-c021b844928c","Type":"ContainerDied","Data":"6e96d0120b2052286ad067f862cf3b9bb071c427dc81e21964d9623c5d07eb36"}
May 28 08:21:54.418959 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.418698    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e96d0120b2052286ad067f862cf3b9bb071c427dc81e21964d9623c5d07eb36"
May 28 08:21:54.420698 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.420672    2551 generic.go:358] "Generic (PLEG): container finished" podID="47b9dd47-358a-4148-a233-d56e3462f334" containerID="7d9bb30e5829dc66f02a19d36a025911817b50b3229917ea187fe6017306291a" exitCode=137
May 28 08:21:54.420805 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.420710    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" event={"ID":"47b9dd47-358a-4148-a233-d56e3462f334","Type":"ContainerDied","Data":"7d9bb30e5829dc66f02a19d36a025911817b50b3229917ea187fe6017306291a"}
May 28 08:21:54.420805 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.420737    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc" event={"ID":"47b9dd47-358a-4148-a233-d56e3462f334","Type":"ContainerDied","Data":"a77cf54981c05ccea8331e0e1bbc93a0390d7512ed158d282b24254d79ca1c2b"}
May 28 08:21:54.420805 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.420752    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a77cf54981c05ccea8331e0e1bbc93a0390d7512ed158d282b24254d79ca1c2b"
May 28 08:21:54.420922 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.420810    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:21:54.424463 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.424447    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:21:54.475911 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.475881    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/de23df0f-3ab4-498f-82bb-c021b844928c-registry-tls\") pod \"de23df0f-3ab4-498f-82bb-c021b844928c\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") "
May 28 08:21:54.475911 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.475913    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/47b9dd47-358a-4148-a233-d56e3462f334-feast-data\") pod \"47b9dd47-358a-4148-a233-d56e3462f334\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") "
May 28 08:21:54.476129 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.475947    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxxjr\" (UniqueName: \"kubernetes.io/projected/de23df0f-3ab4-498f-82bb-c021b844928c-kube-api-access-pxxjr\") pod \"de23df0f-3ab4-498f-82bb-c021b844928c\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") "
May 28 08:21:54.476129 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.475975    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/de23df0f-3ab4-498f-82bb-c021b844928c-online-tls\") pod \"de23df0f-3ab4-498f-82bb-c021b844928c\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") "
May 28 08:21:54.476129 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.476000    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/de23df0f-3ab4-498f-82bb-c021b844928c-feast-data\") pod \"de23df0f-3ab4-498f-82bb-c021b844928c\" (UID: \"de23df0f-3ab4-498f-82bb-c021b844928c\") "
May 28 08:21:54.476129 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.476029    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/47b9dd47-358a-4148-a233-d56e3462f334-registry-tls\") pod \"47b9dd47-358a-4148-a233-d56e3462f334\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") "
May 28 08:21:54.476129 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.476099    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76vnm\" (UniqueName: \"kubernetes.io/projected/47b9dd47-358a-4148-a233-d56e3462f334-kube-api-access-76vnm\") pod \"47b9dd47-358a-4148-a233-d56e3462f334\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") "
May 28 08:21:54.476129 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.476129    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47b9dd47-358a-4148-a233-d56e3462f334-online-tls\") pod \"47b9dd47-358a-4148-a233-d56e3462f334\" (UID: \"47b9dd47-358a-4148-a233-d56e3462f334\") "
May 28 08:21:54.476542 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.476432    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b9dd47-358a-4148-a233-d56e3462f334-feast-data" (OuterVolumeSpecName: "feast-data") pod "47b9dd47-358a-4148-a233-d56e3462f334" (UID: "47b9dd47-358a-4148-a233-d56e3462f334"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 08:21:54.476727 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.476702    2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/47b9dd47-358a-4148-a233-d56e3462f334-feast-data\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:21:54.478714 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.478678    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b9dd47-358a-4148-a233-d56e3462f334-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "47b9dd47-358a-4148-a233-d56e3462f334" (UID: "47b9dd47-358a-4148-a233-d56e3462f334"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:21:54.478883 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.478856    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de23df0f-3ab4-498f-82bb-c021b844928c-kube-api-access-pxxjr" (OuterVolumeSpecName: "kube-api-access-pxxjr") pod "de23df0f-3ab4-498f-82bb-c021b844928c" (UID: "de23df0f-3ab4-498f-82bb-c021b844928c"). InnerVolumeSpecName "kube-api-access-pxxjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:21:54.478980 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.478892    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b9dd47-358a-4148-a233-d56e3462f334-online-tls" (OuterVolumeSpecName: "online-tls") pod "47b9dd47-358a-4148-a233-d56e3462f334" (UID: "47b9dd47-358a-4148-a233-d56e3462f334"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:21:54.478980 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.478892    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de23df0f-3ab4-498f-82bb-c021b844928c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "de23df0f-3ab4-498f-82bb-c021b844928c" (UID: "de23df0f-3ab4-498f-82bb-c021b844928c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:21:54.478980 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.478924    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de23df0f-3ab4-498f-82bb-c021b844928c-online-tls" (OuterVolumeSpecName: "online-tls") pod "de23df0f-3ab4-498f-82bb-c021b844928c" (UID: "de23df0f-3ab4-498f-82bb-c021b844928c"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:21:54.479153 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.479081    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b9dd47-358a-4148-a233-d56e3462f334-kube-api-access-76vnm" (OuterVolumeSpecName: "kube-api-access-76vnm") pod "47b9dd47-358a-4148-a233-d56e3462f334" (UID: "47b9dd47-358a-4148-a233-d56e3462f334"). InnerVolumeSpecName "kube-api-access-76vnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:21:54.487315 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.487291    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de23df0f-3ab4-498f-82bb-c021b844928c-feast-data" (OuterVolumeSpecName: "feast-data") pod "de23df0f-3ab4-498f-82bb-c021b844928c" (UID: "de23df0f-3ab4-498f-82bb-c021b844928c"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 08:21:54.577549 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.577483    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76vnm\" (UniqueName: \"kubernetes.io/projected/47b9dd47-358a-4148-a233-d56e3462f334-kube-api-access-76vnm\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:21:54.577549 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.577508    2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/47b9dd47-358a-4148-a233-d56e3462f334-online-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:21:54.577549 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.577519    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/de23df0f-3ab4-498f-82bb-c021b844928c-registry-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:21:54.577549 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.577527    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxxjr\" (UniqueName: \"kubernetes.io/projected/de23df0f-3ab4-498f-82bb-c021b844928c-kube-api-access-pxxjr\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:21:54.577549 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.577536    2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/de23df0f-3ab4-498f-82bb-c021b844928c-online-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:21:54.577549 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.577544    2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/de23df0f-3ab4-498f-82bb-c021b844928c-feast-data\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:21:54.577549 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:54.577552    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/47b9dd47-358a-4148-a233-d56e3462f334-registry-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:21:55.423524 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:55.423493    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"
May 28 08:21:55.423900 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:55.423526    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"
May 28 08:21:55.445869 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:55.445833    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"]
May 28 08:21:55.448399 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:55.448360    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-5b9745568c-9hg59"]
May 28 08:21:55.457715 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:55.457696    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"]
May 28 08:21:55.461159 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:55.461136    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-7f654c97b-g4ftc"]
May 28 08:21:56.195836 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:56.195802    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47b9dd47-358a-4148-a233-d56e3462f334" path="/var/lib/kubelet/pods/47b9dd47-358a-4148-a233-d56e3462f334/volumes"
May 28 08:21:56.196601 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:21:56.196575    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" path="/var/lib/kubelet/pods/de23df0f-3ab4-498f-82bb-c021b844928c/volumes"
May 28 08:22:00.450993 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.450959    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"]
May 28 08:22:00.451524 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451440    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="feast-init"
May 28 08:22:00.451524 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451460    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="feast-init"
May 28 08:22:00.451524 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451473    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="online"
May 28 08:22:00.451524 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451499    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="online"
May 28 08:22:00.451524 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451522    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="registry"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451530    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="registry"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451541    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="online"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451550    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="online"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451560    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="feast-init"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451569    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="feast-init"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451585    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="registry"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451593    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="registry"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451606    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="feast-apply"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451614    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="feast-apply"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451622    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="feast-apply"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451633    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="feast-apply"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451645    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d80bd210-9c9b-478e-9c52-ab9a8317393f" containerName="manager"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451652    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d80bd210-9c9b-478e-9c52-ab9a8317393f" containerName="manager"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451740    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="online"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451755    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="registry"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451769    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="d80bd210-9c9b-478e-9c52-ab9a8317393f" containerName="manager"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451780    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="de23df0f-3ab4-498f-82bb-c021b844928c" containerName="online"
May 28 08:22:00.451823 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.451792    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="47b9dd47-358a-4148-a233-d56e3462f334" containerName="registry"
May 28 08:22:00.456506 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.456484    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:00.459114 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.458803    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 28 08:22:00.459114 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.458833    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-kw9m9\""
May 28 08:22:00.459114 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.458896    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 28 08:22:00.459114 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.459023    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 28 08:22:00.459114 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.459029    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 28 08:22:00.460059 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.459257    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 08:22:00.460059 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.459266    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 08:22:00.466827 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.465859    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"]
May 28 08:22:00.526932 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.526899    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sd8l\" (UniqueName: \"kubernetes.io/projected/84e3b1fb-efea-4f0c-9a59-c361897bfd86-kube-api-access-2sd8l\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:00.527097 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.526944    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:00.527097 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.527029    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:00.527097 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.527091    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/84e3b1fb-efea-4f0c-9a59-c361897bfd86-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:00.527267 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.527134    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:00.527267 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.527177    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:00.628286 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.628252    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2sd8l\" (UniqueName: \"kubernetes.io/projected/84e3b1fb-efea-4f0c-9a59-c361897bfd86-kube-api-access-2sd8l\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:00.628493 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.628301    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:00.628493 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.628342    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:00.628493 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.628451    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/84e3b1fb-efea-4f0c-9a59-c361897bfd86-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:00.628697 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.628500    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:00.628697 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.628562    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:00.628697 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:22:00.628463    2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 28 08:22:00.628697 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:22:00.628657    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-ui-tls podName:84e3b1fb-efea-4f0c-9a59-c361897bfd86 nodeName:}" failed. No retries permitted until 2026-05-28 08:22:01.128635704 +0000 UTC m=+1373.568707975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-ui-tls") pod "feast-simple-feast-setup-664fcfcc58-5gwjn" (UID: "84e3b1fb-efea-4f0c-9a59-c361897bfd86") : secret "feast-simple-feast-setup-ui-tls" not found
May 28 08:22:00.629025 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:22:00.628706    2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 28 08:22:00.629025 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:22:00.628780    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-online-tls podName:84e3b1fb-efea-4f0c-9a59-c361897bfd86 nodeName:}" failed. No retries permitted until 2026-05-28 08:22:01.128762365 +0000 UTC m=+1373.568834639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-online-tls") pod "feast-simple-feast-setup-664fcfcc58-5gwjn" (UID: "84e3b1fb-efea-4f0c-9a59-c361897bfd86") : secret "feast-simple-feast-setup-online-tls" not found
May 28 08:22:00.629025 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:22:00.628867    2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 28 08:22:00.629025 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:22:00.628936    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-offline-tls podName:84e3b1fb-efea-4f0c-9a59-c361897bfd86 nodeName:}" failed. No retries permitted until 2026-05-28 08:22:01.128919076 +0000 UTC m=+1373.568991350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-offline-tls") pod "feast-simple-feast-setup-664fcfcc58-5gwjn" (UID: "84e3b1fb-efea-4f0c-9a59-c361897bfd86") : secret "feast-simple-feast-setup-offline-tls" not found
May 28 08:22:00.629025 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.628867    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/84e3b1fb-efea-4f0c-9a59-c361897bfd86-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:00.631425 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.631377    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:00.637574 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:00.637509    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sd8l\" (UniqueName: \"kubernetes.io/projected/84e3b1fb-efea-4f0c-9a59-c361897bfd86-kube-api-access-2sd8l\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:01.131829 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:01.131788    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:01.131989 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:01.131839    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:01.131989 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:01.131887    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:01.134378 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:01.134346    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:01.134524 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:01.134420    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:01.134524 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:01.134420    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5gwjn\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:01.377614 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:01.377567    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:01.509330 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:01.509307    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"]
May 28 08:22:01.511422 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:22:01.511378    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84e3b1fb_efea_4f0c_9a59_c361897bfd86.slice/crio-0c021f53d655e90592e28cd8f57ffbfbb27b5ecd655c36ceea8959385b2dcc5a WatchSource:0}: Error finding container 0c021f53d655e90592e28cd8f57ffbfbb27b5ecd655c36ceea8959385b2dcc5a: Status 404 returned error can't find the container with id 0c021f53d655e90592e28cd8f57ffbfbb27b5ecd655c36ceea8959385b2dcc5a
May 28 08:22:02.450173 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:02.450136    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" event={"ID":"84e3b1fb-efea-4f0c-9a59-c361897bfd86","Type":"ContainerStarted","Data":"0c021f53d655e90592e28cd8f57ffbfbb27b5ecd655c36ceea8959385b2dcc5a"}
May 28 08:22:18.505635 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:18.505596    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" event={"ID":"84e3b1fb-efea-4f0c-9a59-c361897bfd86","Type":"ContainerStarted","Data":"c89540c9b95311b4b639679e59565b750078bc0c85e9dde3013fc2d423425b2c"}
May 28 08:22:23.521862 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:23.521821    2551 generic.go:358] "Generic (PLEG): container finished" podID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerID="c89540c9b95311b4b639679e59565b750078bc0c85e9dde3013fc2d423425b2c" exitCode=0
May 28 08:22:23.522280 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:23.521900    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" event={"ID":"84e3b1fb-efea-4f0c-9a59-c361897bfd86","Type":"ContainerDied","Data":"c89540c9b95311b4b639679e59565b750078bc0c85e9dde3013fc2d423425b2c"}
May 28 08:22:24.527259 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:24.527223    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" event={"ID":"84e3b1fb-efea-4f0c-9a59-c361897bfd86","Type":"ContainerStarted","Data":"a8517ae466496ce931bc7023cc1483e8bdd36dfb504ecf1c6f969e678b2eb4a4"}
May 28 08:22:29.543404 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:29.543349    2551 generic.go:358] "Generic (PLEG): container finished" podID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerID="a8517ae466496ce931bc7023cc1483e8bdd36dfb504ecf1c6f969e678b2eb4a4" exitCode=0
May 28 08:22:29.543859 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:29.543433    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" event={"ID":"84e3b1fb-efea-4f0c-9a59-c361897bfd86","Type":"ContainerDied","Data":"a8517ae466496ce931bc7023cc1483e8bdd36dfb504ecf1c6f969e678b2eb4a4"}
May 28 08:22:30.551990 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:30.551948    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" event={"ID":"84e3b1fb-efea-4f0c-9a59-c361897bfd86","Type":"ContainerStarted","Data":"301550ae3af77bf0b65101c6ca1d43bbafe1726ae8feaca021e20ac312cd5360"}
May 28 08:22:30.552497 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:30.551999    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" event={"ID":"84e3b1fb-efea-4f0c-9a59-c361897bfd86","Type":"ContainerStarted","Data":"009d3295a0f8e25cc4d3e611c942ec3b66ee6f7c659d826c48d2b602a9cde73e"}
May 28 08:22:30.552497 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:30.552014    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" event={"ID":"84e3b1fb-efea-4f0c-9a59-c361897bfd86","Type":"ContainerStarted","Data":"cdac344a11fae2d6e1de5f412499bae7dd8de62ca8e8c2e207d1dce9c811c8fd"}
May 28 08:22:30.552497 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:30.552031    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" event={"ID":"84e3b1fb-efea-4f0c-9a59-c361897bfd86","Type":"ContainerStarted","Data":"1b63d372a11c4b0dcfcb546b95f83dde29b5cdd00857b4123d0cf576387f9122"}
May 28 08:22:30.575581 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:30.575516    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podStartSLOduration=14.318652045 podStartE2EDuration="30.575497026s" podCreationTimestamp="2026-05-28 08:22:00 +0000 UTC" firstStartedPulling="2026-05-28 08:22:01.513132686 +0000 UTC m=+1373.953204954" lastFinishedPulling="2026-05-28 08:22:17.769977649 +0000 UTC m=+1390.210049935" observedRunningTime="2026-05-28 08:22:30.574185036 +0000 UTC m=+1403.014257331" watchObservedRunningTime="2026-05-28 08:22:30.575497026 +0000 UTC m=+1403.015569315"
May 28 08:22:31.378399 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:31.378127    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:31.378610 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:31.378432    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:31.378610 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:31.378453    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:31.378610 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:31.378468    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:31.378610 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:31.378481    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:31.378610 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:31.378493    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:31.378610 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:31.378506    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:31.378610 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:31.378517    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:31.380635 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:31.380587    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.31:8816: connect: connection refused"
May 28 08:22:31.380779 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:31.380613    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused"
May 28 08:22:31.380779 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:31.380682    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.31:8443: connect: connection refused"
May 28 08:22:31.380779 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:31.380623    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused"
May 28 08:22:34.378824 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:34.378646    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.31:8443: connect: connection refused"
May 28 08:22:34.378824 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:34.378675    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused"
May 28 08:22:34.378824 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:34.378645    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused"
May 28 08:22:34.378824 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:34.378649    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.31:8816: connect: connection refused"
May 28 08:22:37.379185 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:37.379155    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:37.379606 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:37.379285    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:37.379606 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:37.379579    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:37.380128 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:37.380099    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:37.380209 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:37.380162    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:37.380209 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:37.380202    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:37.383602 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:37.383585    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:22:37.580124 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:22:37.580089    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:23:39.811719 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:39.811677    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"]
May 28 08:23:39.812289 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:39.812132    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="registry" containerID="cri-o://1b63d372a11c4b0dcfcb546b95f83dde29b5cdd00857b4123d0cf576387f9122" gracePeriod=30
May 28 08:23:39.812289 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:39.812227    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="offline" containerID="cri-o://009d3295a0f8e25cc4d3e611c942ec3b66ee6f7c659d826c48d2b602a9cde73e" gracePeriod=30
May 28 08:23:39.812289 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:39.812277    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="online" containerID="cri-o://cdac344a11fae2d6e1de5f412499bae7dd8de62ca8e8c2e207d1dce9c811c8fd" gracePeriod=30
May 28 08:23:39.812497 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:39.812357    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="ui" containerID="cri-o://301550ae3af77bf0b65101c6ca1d43bbafe1726ae8feaca021e20ac312cd5360" gracePeriod=30
May 28 08:23:40.165892 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.165784    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"]
May 28 08:23:40.170605 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.170578    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.173567 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.173361    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-2274r\""
May 28 08:23:40.185333 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.185248    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"]
May 28 08:23:40.277074 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.277037    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c7569e81-9e55-4dce-b346-69a7a400546f-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.277213 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.277118    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.277280 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.277207    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.277327 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.277275    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.277435 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.277344    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.277489 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.277433    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-676rr\" (UniqueName: \"kubernetes.io/projected/c7569e81-9e55-4dce-b346-69a7a400546f-kube-api-access-676rr\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.378368 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.378326    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.378368 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.378371    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.378657 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.378424    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.378657 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.378446    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.378657 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.378469    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-676rr\" (UniqueName: \"kubernetes.io/projected/c7569e81-9e55-4dce-b346-69a7a400546f-kube-api-access-676rr\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.378657 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.378518    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c7569e81-9e55-4dce-b346-69a7a400546f-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.378657 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:23:40.378571    2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 28 08:23:40.378657 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:23:40.378643    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-online-tls podName:c7569e81-9e55-4dce-b346-69a7a400546f nodeName:}" failed. No retries permitted until 2026-05-28 08:23:40.878622252 +0000 UTC m=+1473.318694520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-online-tls") pod "feast-simple-feast-setup-664fcfcc58-8q665" (UID: "c7569e81-9e55-4dce-b346-69a7a400546f") : secret "feast-simple-feast-setup-online-tls" not found
May 28 08:23:40.379035 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.379009    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c7569e81-9e55-4dce-b346-69a7a400546f-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.381145 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.381114    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.381491 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.381469    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.381547 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.381507    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.386364 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.386340    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-676rr\" (UniqueName: \"kubernetes.io/projected/c7569e81-9e55-4dce-b346-69a7a400546f-kube-api-access-676rr\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.780710 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.780626    2551 generic.go:358] "Generic (PLEG): container finished" podID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerID="301550ae3af77bf0b65101c6ca1d43bbafe1726ae8feaca021e20ac312cd5360" exitCode=0
May 28 08:23:40.780710 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.780650    2551 generic.go:358] "Generic (PLEG): container finished" podID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerID="cdac344a11fae2d6e1de5f412499bae7dd8de62ca8e8c2e207d1dce9c811c8fd" exitCode=0
May 28 08:23:40.780710 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.780670    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" event={"ID":"84e3b1fb-efea-4f0c-9a59-c361897bfd86","Type":"ContainerDied","Data":"301550ae3af77bf0b65101c6ca1d43bbafe1726ae8feaca021e20ac312cd5360"}
May 28 08:23:40.780710 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.780692    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" event={"ID":"84e3b1fb-efea-4f0c-9a59-c361897bfd86","Type":"ContainerDied","Data":"cdac344a11fae2d6e1de5f412499bae7dd8de62ca8e8c2e207d1dce9c811c8fd"}
May 28 08:23:40.883309 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.883274    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:40.885650 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:40.885626    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8q665\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:41.085655 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:41.085558    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:41.214891 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:41.214842    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"]
May 28 08:23:41.217034 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:23:41.217005    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7569e81_9e55_4dce_b346_69a7a400546f.slice/crio-76bb93da211e6fda728294b87aa92344dba77c1b73dd0628931e75807e4d6ac3 WatchSource:0}: Error finding container 76bb93da211e6fda728294b87aa92344dba77c1b73dd0628931e75807e4d6ac3: Status 404 returned error can't find the container with id 76bb93da211e6fda728294b87aa92344dba77c1b73dd0628931e75807e4d6ac3
May 28 08:23:41.785439 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:41.785379    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" event={"ID":"c7569e81-9e55-4dce-b346-69a7a400546f","Type":"ContainerStarted","Data":"4a04490694934bf5a67b87cd761efc29fb4553d108af7064af9e43426b95c788"}
May 28 08:23:41.785439 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:41.785444    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" event={"ID":"c7569e81-9e55-4dce-b346-69a7a400546f","Type":"ContainerStarted","Data":"76bb93da211e6fda728294b87aa92344dba77c1b73dd0628931e75807e4d6ac3"}
May 28 08:23:45.800639 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:45.800606    2551 generic.go:358] "Generic (PLEG): container finished" podID="c7569e81-9e55-4dce-b346-69a7a400546f" containerID="4a04490694934bf5a67b87cd761efc29fb4553d108af7064af9e43426b95c788" exitCode=0
May 28 08:23:45.800969 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:45.800674    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" event={"ID":"c7569e81-9e55-4dce-b346-69a7a400546f","Type":"ContainerDied","Data":"4a04490694934bf5a67b87cd761efc29fb4553d108af7064af9e43426b95c788"}
May 28 08:23:46.808471 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:46.808435    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" event={"ID":"c7569e81-9e55-4dce-b346-69a7a400546f","Type":"ContainerStarted","Data":"468c652895a965b66fe4464e20238fdc4c0b4f6d6a2b52197721c8f3413f6178"}
May 28 08:23:47.380083 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:47.380027    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.31:8443: connect: connection refused"
May 28 08:23:47.577587 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:47.577546    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused"
May 28 08:23:51.826047 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:51.826006    2551 generic.go:358] "Generic (PLEG): container finished" podID="c7569e81-9e55-4dce-b346-69a7a400546f" containerID="468c652895a965b66fe4464e20238fdc4c0b4f6d6a2b52197721c8f3413f6178" exitCode=0
May 28 08:23:51.826446 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:51.826077    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" event={"ID":"c7569e81-9e55-4dce-b346-69a7a400546f","Type":"ContainerDied","Data":"468c652895a965b66fe4464e20238fdc4c0b4f6d6a2b52197721c8f3413f6178"}
May 28 08:23:52.833801 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:52.833758    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" event={"ID":"c7569e81-9e55-4dce-b346-69a7a400546f","Type":"ContainerStarted","Data":"6bec7334edf70c36c8e5822baf70ecb6d8f67c44ae46c20532ea0876234ebd5d"}
May 28 08:23:52.833801 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:52.833806    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" event={"ID":"c7569e81-9e55-4dce-b346-69a7a400546f","Type":"ContainerStarted","Data":"48b2e94ef0643571bd7de55057c171410b53b225add8035c6cbde01c9c0ad6f6"}
May 28 08:23:52.834343 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:52.833823    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" event={"ID":"c7569e81-9e55-4dce-b346-69a7a400546f","Type":"ContainerStarted","Data":"b6734ad57efaec4d108ce95a9759d3b266db08a9da03bd1aa3f790064c3070d3"}
May 28 08:23:52.834343 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:52.833845    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" event={"ID":"c7569e81-9e55-4dce-b346-69a7a400546f","Type":"ContainerStarted","Data":"7677f94c2251451c8381e0dc05c6a57225409938ca65609e8763303bcaf8b883"}
May 28 08:23:52.857165 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:52.857089    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podStartSLOduration=12.857070656 podStartE2EDuration="12.857070656s" podCreationTimestamp="2026-05-28 08:23:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:23:52.855434458 +0000 UTC m=+1485.295506748" watchObservedRunningTime="2026-05-28 08:23:52.857070656 +0000 UTC m=+1485.297142946"
May 28 08:23:53.086082 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:53.085981    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:53.086082 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:53.086039    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:53.086082 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:53.086055    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:53.086082 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:53.086068    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:53.088204 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:53.088161    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused"
May 28 08:23:53.088346 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:53.088199    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.32:8816: connect: connection refused"
May 28 08:23:53.088346 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:53.088213    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 28 08:23:53.088346 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:53.088268    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused"
May 28 08:23:56.085969 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:56.085915    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused"
May 28 08:23:56.086499 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:56.085923    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 28 08:23:56.086499 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:56.085923    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused"
May 28 08:23:56.086499 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:56.085940    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.32:8816: connect: connection refused"
May 28 08:23:57.380932 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:57.380879    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.31:8443: connect: connection refused"
May 28 08:23:57.577403 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:57.577347    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused"
May 28 08:23:59.087230 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:59.087189    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:59.087630 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:59.087531    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:59.087630 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:59.087622    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:59.088142 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:59.088110    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:59.088259 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:59.088150    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:59.088259 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:59.088164    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:59.088361 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:59.088342    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:59.088494 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:59.088477    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:59.088668 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:59.088657    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:59.091706 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:59.091691    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:59.859050 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:59.859020    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:23:59.861969 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:23:59.861945    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:24:07.380283 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:07.380240    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.31:8443: connect: connection refused"
May 28 08:24:07.380726 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:07.380438    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:24:07.578284 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:07.578242    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused"
May 28 08:24:07.578484 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:07.578366    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:24:08.185019 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:08.184987    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:24:08.185949 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:08.185926    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:24:09.891434 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:09.891377    2551 generic.go:358] "Generic (PLEG): container finished" podID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerID="009d3295a0f8e25cc4d3e611c942ec3b66ee6f7c659d826c48d2b602a9cde73e" exitCode=137
May 28 08:24:09.891434 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:09.891414    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" event={"ID":"84e3b1fb-efea-4f0c-9a59-c361897bfd86","Type":"ContainerDied","Data":"009d3295a0f8e25cc4d3e611c942ec3b66ee6f7c659d826c48d2b602a9cde73e"}
May 28 08:24:09.891791 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:09.891453    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" event={"ID":"84e3b1fb-efea-4f0c-9a59-c361897bfd86","Type":"ContainerDied","Data":"1b63d372a11c4b0dcfcb546b95f83dde29b5cdd00857b4123d0cf576387f9122"}
May 28 08:24:09.891791 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:09.891425    2551 generic.go:358] "Generic (PLEG): container finished" podID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerID="1b63d372a11c4b0dcfcb546b95f83dde29b5cdd00857b4123d0cf576387f9122" exitCode=137
May 28 08:24:10.460542 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.460519    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:24:10.542649 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.542619    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-online-tls\") pod \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") "
May 28 08:24:10.542811 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.542659    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sd8l\" (UniqueName: \"kubernetes.io/projected/84e3b1fb-efea-4f0c-9a59-c361897bfd86-kube-api-access-2sd8l\") pod \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") "
May 28 08:24:10.542811 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.542704    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-offline-tls\") pod \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") "
May 28 08:24:10.542811 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.542721    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/84e3b1fb-efea-4f0c-9a59-c361897bfd86-feast-data\") pod \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") "
May 28 08:24:10.542811 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.542745    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-registry-tls\") pod \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") "
May 28 08:24:10.542811 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.542768    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-ui-tls\") pod \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\" (UID: \"84e3b1fb-efea-4f0c-9a59-c361897bfd86\") "
May 28 08:24:10.543374 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.543338    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e3b1fb-efea-4f0c-9a59-c361897bfd86-feast-data" (OuterVolumeSpecName: "feast-data") pod "84e3b1fb-efea-4f0c-9a59-c361897bfd86" (UID: "84e3b1fb-efea-4f0c-9a59-c361897bfd86"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 08:24:10.545009 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.544975    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "84e3b1fb-efea-4f0c-9a59-c361897bfd86" (UID: "84e3b1fb-efea-4f0c-9a59-c361897bfd86"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:24:10.545009 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.544999    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-online-tls" (OuterVolumeSpecName: "online-tls") pod "84e3b1fb-efea-4f0c-9a59-c361897bfd86" (UID: "84e3b1fb-efea-4f0c-9a59-c361897bfd86"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:24:10.545194 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.545054    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e3b1fb-efea-4f0c-9a59-c361897bfd86-kube-api-access-2sd8l" (OuterVolumeSpecName: "kube-api-access-2sd8l") pod "84e3b1fb-efea-4f0c-9a59-c361897bfd86" (UID: "84e3b1fb-efea-4f0c-9a59-c361897bfd86"). InnerVolumeSpecName "kube-api-access-2sd8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:24:10.545194 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.545163    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "84e3b1fb-efea-4f0c-9a59-c361897bfd86" (UID: "84e3b1fb-efea-4f0c-9a59-c361897bfd86"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:24:10.545309 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.545210    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "84e3b1fb-efea-4f0c-9a59-c361897bfd86" (UID: "84e3b1fb-efea-4f0c-9a59-c361897bfd86"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:24:10.643668 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.643579    2551 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-offline-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:24:10.643668 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.643614    2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/84e3b1fb-efea-4f0c-9a59-c361897bfd86-feast-data\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:24:10.643668 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.643623    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-registry-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:24:10.643668 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.643632    2551 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-ui-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:24:10.643668 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.643639    2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/84e3b1fb-efea-4f0c-9a59-c361897bfd86-online-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:24:10.643668 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.643649    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sd8l\" (UniqueName: \"kubernetes.io/projected/84e3b1fb-efea-4f0c-9a59-c361897bfd86-kube-api-access-2sd8l\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:24:10.896981 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.896917    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"
May 28 08:24:10.897457 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.896915    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn" event={"ID":"84e3b1fb-efea-4f0c-9a59-c361897bfd86","Type":"ContainerDied","Data":"0c021f53d655e90592e28cd8f57ffbfbb27b5ecd655c36ceea8959385b2dcc5a"}
May 28 08:24:10.897457 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.897032    2551 scope.go:117] "RemoveContainer" containerID="301550ae3af77bf0b65101c6ca1d43bbafe1726ae8feaca021e20ac312cd5360"
May 28 08:24:10.905847 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.905822    2551 scope.go:117] "RemoveContainer" containerID="009d3295a0f8e25cc4d3e611c942ec3b66ee6f7c659d826c48d2b602a9cde73e"
May 28 08:24:10.913457 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.913412    2551 scope.go:117] "RemoveContainer" containerID="cdac344a11fae2d6e1de5f412499bae7dd8de62ca8e8c2e207d1dce9c811c8fd"
May 28 08:24:10.918328 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.918306    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"]
May 28 08:24:10.921591 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.921574    2551 scope.go:117] "RemoveContainer" containerID="1b63d372a11c4b0dcfcb546b95f83dde29b5cdd00857b4123d0cf576387f9122"
May 28 08:24:10.921668 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.921610    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5gwjn"]
May 28 08:24:10.928394 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.928367    2551 scope.go:117] "RemoveContainer" containerID="a8517ae466496ce931bc7023cc1483e8bdd36dfb504ecf1c6f969e678b2eb4a4"
May 28 08:24:10.935048 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:10.935030    2551 scope.go:117] "RemoveContainer" containerID="c89540c9b95311b4b639679e59565b750078bc0c85e9dde3013fc2d423425b2c"
May 28 08:24:12.194760 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:24:12.194726    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" path="/var/lib/kubelet/pods/84e3b1fb-efea-4f0c-9a59-c361897bfd86/volumes"
May 28 08:25:01.943608 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.943570    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"]
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.943880    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="feast-init"
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.943891    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="feast-init"
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.943903    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="feast-apply"
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.943908    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="feast-apply"
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.943936    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="online"
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.943942    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="online"
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.943948    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="ui"
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.943954    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="ui"
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.943964    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="registry"
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.943969    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="registry"
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.943978    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="offline"
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.943984    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="offline"
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.944031    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="offline"
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.944043    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="registry"
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.944050    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="ui"
May 28 08:25:01.944109 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.944055    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="84e3b1fb-efea-4f0c-9a59-c361897bfd86" containerName="online"
May 28 08:25:01.947346 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.947329    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:01.949857 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.949823    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 28 08:25:01.950062 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.950041    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 28 08:25:01.950175 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.950140    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 28 08:25:01.950270 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.950254    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 28 08:25:01.951147 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.951126    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 28 08:25:01.951251 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.951169    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 28 08:25:01.951251 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.951165    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-vpbbh\""
May 28 08:25:01.958679 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:01.958658    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"]
May 28 08:25:02.076713 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.076678    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.076713 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.076714    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f8fc5968-cb7a-4881-9279-028814014f1e-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.076937 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.076734    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f8fc5968-cb7a-4881-9279-028814014f1e-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.076937 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.076875    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8j4p\" (UniqueName: \"kubernetes.io/projected/f8fc5968-cb7a-4881-9279-028814014f1e-kube-api-access-g8j4p\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.076937 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.076910    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.076937 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.076931    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.178054 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.178025    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.178054 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.178056    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f8fc5968-cb7a-4881-9279-028814014f1e-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.178289 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.178076    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f8fc5968-cb7a-4881-9279-028814014f1e-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.178289 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.178141    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g8j4p\" (UniqueName: \"kubernetes.io/projected/f8fc5968-cb7a-4881-9279-028814014f1e-kube-api-access-g8j4p\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.178289 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.178169    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.178289 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.178193    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.178632 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.178534    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f8fc5968-cb7a-4881-9279-028814014f1e-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.178930 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.178908    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f8fc5968-cb7a-4881-9279-028814014f1e-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.180733 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.180707    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.180829 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.180767    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.180829 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.180797    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.185768 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.185750    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8j4p\" (UniqueName: \"kubernetes.io/projected/f8fc5968-cb7a-4881-9279-028814014f1e-kube-api-access-g8j4p\") pod \"feast-simple-feast-remote-setup-5cff94746b-rplwz\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.259241 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.259210    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:02.388936 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.388906    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"]
May 28 08:25:02.391016 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:25:02.390985    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8fc5968_cb7a_4881_9279_028814014f1e.slice/crio-7491e137a0c77d0034513e52b298c1630c13bb8af0d71c9bed77fb81e668b236 WatchSource:0}: Error finding container 7491e137a0c77d0034513e52b298c1630c13bb8af0d71c9bed77fb81e668b236: Status 404 returned error can't find the container with id 7491e137a0c77d0034513e52b298c1630c13bb8af0d71c9bed77fb81e668b236
May 28 08:25:02.392571 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:02.392554    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 08:25:03.056454 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:03.056415    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" event={"ID":"f8fc5968-cb7a-4881-9279-028814014f1e","Type":"ContainerStarted","Data":"d6e4d84170be192fbd2e74faa3f0a64d2a2f9a4cf2f7732191381eff36d87382"}
May 28 08:25:03.056454 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:03.056455    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" event={"ID":"f8fc5968-cb7a-4881-9279-028814014f1e","Type":"ContainerStarted","Data":"7491e137a0c77d0034513e52b298c1630c13bb8af0d71c9bed77fb81e668b236"}
May 28 08:25:07.071163 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:07.071083    2551 generic.go:358] "Generic (PLEG): container finished" podID="f8fc5968-cb7a-4881-9279-028814014f1e" containerID="d6e4d84170be192fbd2e74faa3f0a64d2a2f9a4cf2f7732191381eff36d87382" exitCode=0
May 28 08:25:07.071163 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:07.071143    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" event={"ID":"f8fc5968-cb7a-4881-9279-028814014f1e","Type":"ContainerDied","Data":"d6e4d84170be192fbd2e74faa3f0a64d2a2f9a4cf2f7732191381eff36d87382"}
May 28 08:25:08.075571 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:08.075532    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" event={"ID":"f8fc5968-cb7a-4881-9279-028814014f1e","Type":"ContainerStarted","Data":"c03c6ffc333d2d6f51a8c22934a13de50138a8b670c0a6fa651be5c8208f3e14"}
May 28 08:25:13.091192 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:13.091159    2551 generic.go:358] "Generic (PLEG): container finished" podID="f8fc5968-cb7a-4881-9279-028814014f1e" containerID="c03c6ffc333d2d6f51a8c22934a13de50138a8b670c0a6fa651be5c8208f3e14" exitCode=0
May 28 08:25:13.091621 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:13.091240    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" event={"ID":"f8fc5968-cb7a-4881-9279-028814014f1e","Type":"ContainerDied","Data":"c03c6ffc333d2d6f51a8c22934a13de50138a8b670c0a6fa651be5c8208f3e14"}
May 28 08:25:14.105965 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:14.105917    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" event={"ID":"f8fc5968-cb7a-4881-9279-028814014f1e","Type":"ContainerStarted","Data":"ac3b8fa03c9dae58413b323fbabc3013714cfdadc099f19a1cd8dff9b24a3fb5"}
May 28 08:25:14.106342 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:14.105977    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" event={"ID":"f8fc5968-cb7a-4881-9279-028814014f1e","Type":"ContainerStarted","Data":"93eb81df78be68ace4a7f7f354f147987a329a67d6873a45c5477441b0a1fdd2"}
May 28 08:25:14.106342 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:14.105994    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" event={"ID":"f8fc5968-cb7a-4881-9279-028814014f1e","Type":"ContainerStarted","Data":"40bd28efb796b3361f95f5aac85abfd48a6bcf9c890d3591569a86897bbc5466"}
May 28 08:25:14.128280 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:14.128226    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podStartSLOduration=13.128207304 podStartE2EDuration="13.128207304s" podCreationTimestamp="2026-05-28 08:25:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:25:14.126649939 +0000 UTC m=+1566.566722224" watchObservedRunningTime="2026-05-28 08:25:14.128207304 +0000 UTC m=+1566.568279597"
May 28 08:25:14.259965 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:14.259911    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:14.259965 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:14.259954    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:14.259965 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:14.259966    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:14.261981 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:14.261937    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused"
May 28 08:25:14.262238 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:14.261956    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.33:8816: connect: connection refused"
May 28 08:25:14.262331 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:14.261976    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 28 08:25:17.260186 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:17.260125    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.33:8816: connect: connection refused"
May 28 08:25:17.260707 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:17.260153    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused"
May 28 08:25:17.260707 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:17.260244    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 28 08:25:20.261180 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:20.261150    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:20.261657 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:20.261234    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:20.261657 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:20.261599    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:20.261657 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:20.261636    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:20.261998 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:20.261977    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:20.262105 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:20.262090    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:20.265073 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:20.265058    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:21.132564 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:21.132532    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:25:21.135414 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:25:21.135378    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:26:08.626700 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:08.626667    2551 scope.go:117] "RemoveContainer" containerID="95d93a1974e6c6ae1bafa4e6b5b4ed25235623df117f9e68f9c00c0d1761747f"
May 28 08:26:08.634572 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:08.634547    2551 scope.go:117] "RemoveContainer" containerID="a15f108d1d89945afa9de7a8caaaa86e85be783175565d445b99f37d122ee72e"
May 28 08:26:08.641782 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:08.641755    2551 scope.go:117] "RemoveContainer" containerID="e96eedbceed0f7c66a8aea20c84d057b4539725f4b81bf3dd7a384a199632bbc"
May 28 08:26:08.648497 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:08.648480    2551 scope.go:117] "RemoveContainer" containerID="78ed9d07892654416fe6a74fd433dba5257d62153de503859ba25838bb4d606a"
May 28 08:26:08.655209 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:08.655190    2551 scope.go:117] "RemoveContainer" containerID="1a4bb543c33577e7b4befa0e78ea4cf362b403490b1b15e5e1a8710dd7dd74b8"
May 28 08:26:08.661606 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:08.661590    2551 scope.go:117] "RemoveContainer" containerID="95c8260385612c79b43db47b39a57eb4da0b4ad3334c6a54dfba532034c6eee8"
May 28 08:26:08.668162 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:08.668139    2551 scope.go:117] "RemoveContainer" containerID="e52f68f74c93b293fca8c157f1a1aea1c52b5362325c15b2260c7c65a60d7f91"
May 28 08:26:08.675512 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:08.675493    2551 scope.go:117] "RemoveContainer" containerID="7d9bb30e5829dc66f02a19d36a025911817b50b3229917ea187fe6017306291a"
May 28 08:26:23.076525 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:23.076483    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"]
May 28 08:26:23.076981 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:23.076917    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="online" containerID="cri-o://40bd28efb796b3361f95f5aac85abfd48a6bcf9c890d3591569a86897bbc5466" gracePeriod=30
May 28 08:26:23.077080 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:23.077059    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="offline" containerID="cri-o://93eb81df78be68ace4a7f7f354f147987a329a67d6873a45c5477441b0a1fdd2" gracePeriod=30
May 28 08:26:23.077164 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:23.077052    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="ui" containerID="cri-o://ac3b8fa03c9dae58413b323fbabc3013714cfdadc099f19a1cd8dff9b24a3fb5" gracePeriod=30
May 28 08:26:23.246453 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:23.246411    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"]
May 28 08:26:23.247063 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:23.246986    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="registry" containerID="cri-o://7677f94c2251451c8381e0dc05c6a57225409938ca65609e8763303bcaf8b883" gracePeriod=30
May 28 08:26:23.247487 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:23.247460    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="ui" containerID="cri-o://6bec7334edf70c36c8e5822baf70ecb6d8f67c44ae46c20532ea0876234ebd5d" gracePeriod=30
May 28 08:26:23.247589 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:23.247577    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="offline" containerID="cri-o://48b2e94ef0643571bd7de55057c171410b53b225add8035c6cbde01c9c0ad6f6" gracePeriod=30
May 28 08:26:23.247711 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:23.247649    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="online" containerID="cri-o://b6734ad57efaec4d108ce95a9759d3b266db08a9da03bd1aa3f790064c3070d3" gracePeriod=30
May 28 08:26:24.324781 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:24.324750    2551 generic.go:358] "Generic (PLEG): container finished" podID="c7569e81-9e55-4dce-b346-69a7a400546f" containerID="6bec7334edf70c36c8e5822baf70ecb6d8f67c44ae46c20532ea0876234ebd5d" exitCode=0
May 28 08:26:24.324781 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:24.324775    2551 generic.go:358] "Generic (PLEG): container finished" podID="c7569e81-9e55-4dce-b346-69a7a400546f" containerID="b6734ad57efaec4d108ce95a9759d3b266db08a9da03bd1aa3f790064c3070d3" exitCode=0
May 28 08:26:24.325212 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:24.324823    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" event={"ID":"c7569e81-9e55-4dce-b346-69a7a400546f","Type":"ContainerDied","Data":"6bec7334edf70c36c8e5822baf70ecb6d8f67c44ae46c20532ea0876234ebd5d"}
May 28 08:26:24.325212 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:24.324858    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" event={"ID":"c7569e81-9e55-4dce-b346-69a7a400546f","Type":"ContainerDied","Data":"b6734ad57efaec4d108ce95a9759d3b266db08a9da03bd1aa3f790064c3070d3"}
May 28 08:26:24.327009 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:24.326990    2551 generic.go:358] "Generic (PLEG): container finished" podID="f8fc5968-cb7a-4881-9279-028814014f1e" containerID="ac3b8fa03c9dae58413b323fbabc3013714cfdadc099f19a1cd8dff9b24a3fb5" exitCode=0
May 28 08:26:24.327009 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:24.327005    2551 generic.go:358] "Generic (PLEG): container finished" podID="f8fc5968-cb7a-4881-9279-028814014f1e" containerID="40bd28efb796b3361f95f5aac85abfd48a6bcf9c890d3591569a86897bbc5466" exitCode=0
May 28 08:26:24.327155 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:24.327029    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" event={"ID":"f8fc5968-cb7a-4881-9279-028814014f1e","Type":"ContainerDied","Data":"ac3b8fa03c9dae58413b323fbabc3013714cfdadc099f19a1cd8dff9b24a3fb5"}
May 28 08:26:24.327155 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:24.327048    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" event={"ID":"f8fc5968-cb7a-4881-9279-028814014f1e","Type":"ContainerDied","Data":"40bd28efb796b3361f95f5aac85abfd48a6bcf9c890d3591569a86897bbc5466"}
May 28 08:26:29.088461 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:29.088412    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused"
May 28 08:26:29.860053 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:29.860013    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 28 08:26:30.262321 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:30.262277    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused"
May 28 08:26:31.132920 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:31.132878    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 28 08:26:39.089055 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:39.089014    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused"
May 28 08:26:39.859513 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:39.859475    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 28 08:26:40.262637 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:40.262595    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused"
May 28 08:26:41.132721 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:41.132680    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 28 08:26:49.088353 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:49.088305    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.32:8443: connect: connection refused"
May 28 08:26:49.088842 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:49.088475    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:26:49.859669 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:49.859629    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused"
May 28 08:26:49.859831 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:49.859745    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:26:50.262332 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:50.262294    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.33:8443: connect: connection refused"
May 28 08:26:50.262728 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:50.262437    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:26:51.133076 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:51.133035    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused"
May 28 08:26:51.133246 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:51.133158    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:26:53.271728 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.271704    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:26:53.419788 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.419696    2551 generic.go:358] "Generic (PLEG): container finished" podID="f8fc5968-cb7a-4881-9279-028814014f1e" containerID="93eb81df78be68ace4a7f7f354f147987a329a67d6873a45c5477441b0a1fdd2" exitCode=137
May 28 08:26:53.419788 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.419776    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"
May 28 08:26:53.419994 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.419782    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" event={"ID":"f8fc5968-cb7a-4881-9279-028814014f1e","Type":"ContainerDied","Data":"93eb81df78be68ace4a7f7f354f147987a329a67d6873a45c5477441b0a1fdd2"}
May 28 08:26:53.419994 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.419826    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz" event={"ID":"f8fc5968-cb7a-4881-9279-028814014f1e","Type":"ContainerDied","Data":"7491e137a0c77d0034513e52b298c1630c13bb8af0d71c9bed77fb81e668b236"}
May 28 08:26:53.419994 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.419845    2551 scope.go:117] "RemoveContainer" containerID="ac3b8fa03c9dae58413b323fbabc3013714cfdadc099f19a1cd8dff9b24a3fb5"
May 28 08:26:53.422476 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.422455    2551 generic.go:358] "Generic (PLEG): container finished" podID="c7569e81-9e55-4dce-b346-69a7a400546f" containerID="48b2e94ef0643571bd7de55057c171410b53b225add8035c6cbde01c9c0ad6f6" exitCode=137
May 28 08:26:53.422476 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.422476    2551 generic.go:358] "Generic (PLEG): container finished" podID="c7569e81-9e55-4dce-b346-69a7a400546f" containerID="7677f94c2251451c8381e0dc05c6a57225409938ca65609e8763303bcaf8b883" exitCode=137
May 28 08:26:53.422618 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.422509    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" event={"ID":"c7569e81-9e55-4dce-b346-69a7a400546f","Type":"ContainerDied","Data":"48b2e94ef0643571bd7de55057c171410b53b225add8035c6cbde01c9c0ad6f6"}
May 28 08:26:53.422618 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.422531    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" event={"ID":"c7569e81-9e55-4dce-b346-69a7a400546f","Type":"ContainerDied","Data":"7677f94c2251451c8381e0dc05c6a57225409938ca65609e8763303bcaf8b883"}
May 28 08:26:53.428270 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.428255    2551 scope.go:117] "RemoveContainer" containerID="93eb81df78be68ace4a7f7f354f147987a329a67d6873a45c5477441b0a1fdd2"
May 28 08:26:53.435559 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.435544    2551 scope.go:117] "RemoveContainer" containerID="40bd28efb796b3361f95f5aac85abfd48a6bcf9c890d3591569a86897bbc5466"
May 28 08:26:53.440570 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.440551    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-ui-tls\") pod \"f8fc5968-cb7a-4881-9279-028814014f1e\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") "
May 28 08:26:53.440649 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.440587    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f8fc5968-cb7a-4881-9279-028814014f1e-registry-tls\") pod \"f8fc5968-cb7a-4881-9279-028814014f1e\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") "
May 28 08:26:53.440649 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.440613    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f8fc5968-cb7a-4881-9279-028814014f1e-feast-data\") pod \"f8fc5968-cb7a-4881-9279-028814014f1e\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") "
May 28 08:26:53.440775 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.440661    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-online-tls\") pod \"f8fc5968-cb7a-4881-9279-028814014f1e\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") "
May 28 08:26:53.440775 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.440756    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8j4p\" (UniqueName: \"kubernetes.io/projected/f8fc5968-cb7a-4881-9279-028814014f1e-kube-api-access-g8j4p\") pod \"f8fc5968-cb7a-4881-9279-028814014f1e\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") "
May 28 08:26:53.440882 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.440807    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-offline-tls\") pod \"f8fc5968-cb7a-4881-9279-028814014f1e\" (UID: \"f8fc5968-cb7a-4881-9279-028814014f1e\") "
May 28 08:26:53.441142 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.440951    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8fc5968-cb7a-4881-9279-028814014f1e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f8fc5968-cb7a-4881-9279-028814014f1e" (UID: "f8fc5968-cb7a-4881-9279-028814014f1e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 08:26:53.441142 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.441069    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8fc5968-cb7a-4881-9279-028814014f1e-feast-data" (OuterVolumeSpecName: "feast-data") pod "f8fc5968-cb7a-4881-9279-028814014f1e" (UID: "f8fc5968-cb7a-4881-9279-028814014f1e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 08:26:53.441142 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.441109    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/f8fc5968-cb7a-4881-9279-028814014f1e-registry-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:26:53.442932 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.442901    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "f8fc5968-cb7a-4881-9279-028814014f1e" (UID: "f8fc5968-cb7a-4881-9279-028814014f1e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:26:53.443079 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.443060    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "f8fc5968-cb7a-4881-9279-028814014f1e" (UID: "f8fc5968-cb7a-4881-9279-028814014f1e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:26:53.443145 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.443107    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-online-tls" (OuterVolumeSpecName: "online-tls") pod "f8fc5968-cb7a-4881-9279-028814014f1e" (UID: "f8fc5968-cb7a-4881-9279-028814014f1e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:26:53.443145 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.443108    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8fc5968-cb7a-4881-9279-028814014f1e-kube-api-access-g8j4p" (OuterVolumeSpecName: "kube-api-access-g8j4p") pod "f8fc5968-cb7a-4881-9279-028814014f1e" (UID: "f8fc5968-cb7a-4881-9279-028814014f1e"). InnerVolumeSpecName "kube-api-access-g8j4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:26:53.443293 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.443230    2551 scope.go:117] "RemoveContainer" containerID="c03c6ffc333d2d6f51a8c22934a13de50138a8b670c0a6fa651be5c8208f3e14"
May 28 08:26:53.453551 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.453534    2551 scope.go:117] "RemoveContainer" containerID="d6e4d84170be192fbd2e74faa3f0a64d2a2f9a4cf2f7732191381eff36d87382"
May 28 08:26:53.459961 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.459943    2551 scope.go:117] "RemoveContainer" containerID="ac3b8fa03c9dae58413b323fbabc3013714cfdadc099f19a1cd8dff9b24a3fb5"
May 28 08:26:53.460230 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:26:53.460213    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3b8fa03c9dae58413b323fbabc3013714cfdadc099f19a1cd8dff9b24a3fb5\": container with ID starting with ac3b8fa03c9dae58413b323fbabc3013714cfdadc099f19a1cd8dff9b24a3fb5 not found: ID does not exist" containerID="ac3b8fa03c9dae58413b323fbabc3013714cfdadc099f19a1cd8dff9b24a3fb5"
May 28 08:26:53.460299 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.460245    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3b8fa03c9dae58413b323fbabc3013714cfdadc099f19a1cd8dff9b24a3fb5"} err="failed to get container status \"ac3b8fa03c9dae58413b323fbabc3013714cfdadc099f19a1cd8dff9b24a3fb5\": rpc error: code = NotFound desc = could not find container \"ac3b8fa03c9dae58413b323fbabc3013714cfdadc099f19a1cd8dff9b24a3fb5\": container with ID starting with ac3b8fa03c9dae58413b323fbabc3013714cfdadc099f19a1cd8dff9b24a3fb5 not found: ID does not exist"
May 28 08:26:53.460299 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.460272    2551 scope.go:117] "RemoveContainer" containerID="93eb81df78be68ace4a7f7f354f147987a329a67d6873a45c5477441b0a1fdd2"
May 28 08:26:53.460527 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:26:53.460509    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93eb81df78be68ace4a7f7f354f147987a329a67d6873a45c5477441b0a1fdd2\": container with ID starting with 93eb81df78be68ace4a7f7f354f147987a329a67d6873a45c5477441b0a1fdd2 not found: ID does not exist" containerID="93eb81df78be68ace4a7f7f354f147987a329a67d6873a45c5477441b0a1fdd2"
May 28 08:26:53.460568 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.460535    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93eb81df78be68ace4a7f7f354f147987a329a67d6873a45c5477441b0a1fdd2"} err="failed to get container status \"93eb81df78be68ace4a7f7f354f147987a329a67d6873a45c5477441b0a1fdd2\": rpc error: code = NotFound desc = could not find container \"93eb81df78be68ace4a7f7f354f147987a329a67d6873a45c5477441b0a1fdd2\": container with ID starting with 93eb81df78be68ace4a7f7f354f147987a329a67d6873a45c5477441b0a1fdd2 not found: ID does not exist"
May 28 08:26:53.460568 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.460552    2551 scope.go:117] "RemoveContainer" containerID="40bd28efb796b3361f95f5aac85abfd48a6bcf9c890d3591569a86897bbc5466"
May 28 08:26:53.460720 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:26:53.460700    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40bd28efb796b3361f95f5aac85abfd48a6bcf9c890d3591569a86897bbc5466\": container with ID starting with 40bd28efb796b3361f95f5aac85abfd48a6bcf9c890d3591569a86897bbc5466 not found: ID does not exist" containerID="40bd28efb796b3361f95f5aac85abfd48a6bcf9c890d3591569a86897bbc5466"
May 28 08:26:53.460766 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.460727    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bd28efb796b3361f95f5aac85abfd48a6bcf9c890d3591569a86897bbc5466"} err="failed to get container status \"40bd28efb796b3361f95f5aac85abfd48a6bcf9c890d3591569a86897bbc5466\": rpc error: code = NotFound desc = could not find container \"40bd28efb796b3361f95f5aac85abfd48a6bcf9c890d3591569a86897bbc5466\": container with ID starting with 40bd28efb796b3361f95f5aac85abfd48a6bcf9c890d3591569a86897bbc5466 not found: ID does not exist"
May 28 08:26:53.460766 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.460749    2551 scope.go:117] "RemoveContainer" containerID="c03c6ffc333d2d6f51a8c22934a13de50138a8b670c0a6fa651be5c8208f3e14"
May 28 08:26:53.460926 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:26:53.460911    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c03c6ffc333d2d6f51a8c22934a13de50138a8b670c0a6fa651be5c8208f3e14\": container with ID starting with c03c6ffc333d2d6f51a8c22934a13de50138a8b670c0a6fa651be5c8208f3e14 not found: ID does not exist" containerID="c03c6ffc333d2d6f51a8c22934a13de50138a8b670c0a6fa651be5c8208f3e14"
May 28 08:26:53.460966 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.460930    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c03c6ffc333d2d6f51a8c22934a13de50138a8b670c0a6fa651be5c8208f3e14"} err="failed to get container status \"c03c6ffc333d2d6f51a8c22934a13de50138a8b670c0a6fa651be5c8208f3e14\": rpc error: code = NotFound desc = could not find container \"c03c6ffc333d2d6f51a8c22934a13de50138a8b670c0a6fa651be5c8208f3e14\": container with ID starting with c03c6ffc333d2d6f51a8c22934a13de50138a8b670c0a6fa651be5c8208f3e14 not found: ID does not exist"
May 28 08:26:53.460966 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.460943    2551 scope.go:117] "RemoveContainer" containerID="d6e4d84170be192fbd2e74faa3f0a64d2a2f9a4cf2f7732191381eff36d87382"
May 28 08:26:53.461139 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:26:53.461123    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e4d84170be192fbd2e74faa3f0a64d2a2f9a4cf2f7732191381eff36d87382\": container with ID starting with d6e4d84170be192fbd2e74faa3f0a64d2a2f9a4cf2f7732191381eff36d87382 not found: ID does not exist" containerID="d6e4d84170be192fbd2e74faa3f0a64d2a2f9a4cf2f7732191381eff36d87382"
May 28 08:26:53.461179 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.461143    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e4d84170be192fbd2e74faa3f0a64d2a2f9a4cf2f7732191381eff36d87382"} err="failed to get container status \"d6e4d84170be192fbd2e74faa3f0a64d2a2f9a4cf2f7732191381eff36d87382\": rpc error: code = NotFound desc = could not find container \"d6e4d84170be192fbd2e74faa3f0a64d2a2f9a4cf2f7732191381eff36d87382\": container with ID starting with d6e4d84170be192fbd2e74faa3f0a64d2a2f9a4cf2f7732191381eff36d87382 not found: ID does not exist"
May 28 08:26:53.542555 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.542526    2551 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-offline-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:26:53.542555 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.542550    2551 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-ui-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:26:53.542555 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.542558    2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/f8fc5968-cb7a-4881-9279-028814014f1e-feast-data\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:26:53.542754 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.542566    2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/f8fc5968-cb7a-4881-9279-028814014f1e-online-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:26:53.542754 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.542575    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8j4p\" (UniqueName: \"kubernetes.io/projected/f8fc5968-cb7a-4881-9279-028814014f1e-kube-api-access-g8j4p\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:26:53.740642 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.740605    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"]
May 28 08:26:53.745796 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.745760    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-rplwz"]
May 28 08:26:53.891920 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:53.891895    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:26:54.047095 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.047002    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-676rr\" (UniqueName: \"kubernetes.io/projected/c7569e81-9e55-4dce-b346-69a7a400546f-kube-api-access-676rr\") pod \"c7569e81-9e55-4dce-b346-69a7a400546f\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") "
May 28 08:26:54.047095 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.047043    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c7569e81-9e55-4dce-b346-69a7a400546f-feast-data\") pod \"c7569e81-9e55-4dce-b346-69a7a400546f\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") "
May 28 08:26:54.047095 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.047086    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-online-tls\") pod \"c7569e81-9e55-4dce-b346-69a7a400546f\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") "
May 28 08:26:54.047686 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.047120    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-ui-tls\") pod \"c7569e81-9e55-4dce-b346-69a7a400546f\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") "
May 28 08:26:54.047686 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.047156    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-registry-tls\") pod \"c7569e81-9e55-4dce-b346-69a7a400546f\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") "
May 28 08:26:54.047686 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.047180    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-offline-tls\") pod \"c7569e81-9e55-4dce-b346-69a7a400546f\" (UID: \"c7569e81-9e55-4dce-b346-69a7a400546f\") "
May 28 08:26:54.047686 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.047663    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7569e81-9e55-4dce-b346-69a7a400546f-feast-data" (OuterVolumeSpecName: "feast-data") pod "c7569e81-9e55-4dce-b346-69a7a400546f" (UID: "c7569e81-9e55-4dce-b346-69a7a400546f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 08:26:54.049270 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.049236    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7569e81-9e55-4dce-b346-69a7a400546f-kube-api-access-676rr" (OuterVolumeSpecName: "kube-api-access-676rr") pod "c7569e81-9e55-4dce-b346-69a7a400546f" (UID: "c7569e81-9e55-4dce-b346-69a7a400546f"). InnerVolumeSpecName "kube-api-access-676rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:26:54.049433 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.049326    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "c7569e81-9e55-4dce-b346-69a7a400546f" (UID: "c7569e81-9e55-4dce-b346-69a7a400546f"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:26:54.049433 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.049342    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "c7569e81-9e55-4dce-b346-69a7a400546f" (UID: "c7569e81-9e55-4dce-b346-69a7a400546f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:26:54.049717 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.049701    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c7569e81-9e55-4dce-b346-69a7a400546f" (UID: "c7569e81-9e55-4dce-b346-69a7a400546f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:26:54.049784 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.049715    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-online-tls" (OuterVolumeSpecName: "online-tls") pod "c7569e81-9e55-4dce-b346-69a7a400546f" (UID: "c7569e81-9e55-4dce-b346-69a7a400546f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:26:54.148018 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.147988    2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-online-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:26:54.148018 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.148015    2551 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-ui-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:26:54.148018 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.148024    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-registry-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:26:54.148198 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.148032    2551 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c7569e81-9e55-4dce-b346-69a7a400546f-offline-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:26:54.148198 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.148041    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-676rr\" (UniqueName: \"kubernetes.io/projected/c7569e81-9e55-4dce-b346-69a7a400546f-kube-api-access-676rr\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:26:54.148198 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.148051    2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c7569e81-9e55-4dce-b346-69a7a400546f-feast-data\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:26:54.195362 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.195327    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" path="/var/lib/kubelet/pods/f8fc5968-cb7a-4881-9279-028814014f1e/volumes"
May 28 08:26:54.431261 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.431180    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"
May 28 08:26:54.431653 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.431189    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665" event={"ID":"c7569e81-9e55-4dce-b346-69a7a400546f","Type":"ContainerDied","Data":"76bb93da211e6fda728294b87aa92344dba77c1b73dd0628931e75807e4d6ac3"}
May 28 08:26:54.431653 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.431311    2551 scope.go:117] "RemoveContainer" containerID="6bec7334edf70c36c8e5822baf70ecb6d8f67c44ae46c20532ea0876234ebd5d"
May 28 08:26:54.439178 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.439162    2551 scope.go:117] "RemoveContainer" containerID="48b2e94ef0643571bd7de55057c171410b53b225add8035c6cbde01c9c0ad6f6"
May 28 08:26:54.446076 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.446057    2551 scope.go:117] "RemoveContainer" containerID="b6734ad57efaec4d108ce95a9759d3b266db08a9da03bd1aa3f790064c3070d3"
May 28 08:26:54.450046 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.450016    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"]
May 28 08:26:54.453373 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.453352    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-8q665"]
May 28 08:26:54.453837 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.453810    2551 scope.go:117] "RemoveContainer" containerID="7677f94c2251451c8381e0dc05c6a57225409938ca65609e8763303bcaf8b883"
May 28 08:26:54.460259 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.460243    2551 scope.go:117] "RemoveContainer" containerID="468c652895a965b66fe4464e20238fdc4c0b4f6d6a2b52197721c8f3413f6178"
May 28 08:26:54.466689 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:54.466674    2551 scope.go:117] "RemoveContainer" containerID="4a04490694934bf5a67b87cd761efc29fb4553d108af7064af9e43426b95c788"
May 28 08:26:56.194728 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:26:56.194695    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" path="/var/lib/kubelet/pods/c7569e81-9e55-4dce-b346-69a7a400546f/volumes"
May 28 08:27:33.750037 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.749935    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"]
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750468    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="offline"
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750484    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="offline"
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750497    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="feast-init"
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750505    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="feast-init"
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750515    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="feast-apply"
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750523    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="feast-apply"
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750536    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="feast-init"
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750544    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="feast-init"
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750561    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="offline"
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750569    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="offline"
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750579    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="online"
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750587    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="online"
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750600    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="registry"
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750607    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="registry"
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750615    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="online"
May 28 08:27:33.750620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750624    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="online"
May 28 08:27:33.751398 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750635    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="feast-apply"
May 28 08:27:33.751398 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750644    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="feast-apply"
May 28 08:27:33.751398 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750656    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="ui"
May 28 08:27:33.751398 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750665    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="ui"
May 28 08:27:33.751398 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750677    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="ui"
May 28 08:27:33.751398 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750685    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="ui"
May 28 08:27:33.751398 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750748    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="online"
May 28 08:27:33.751398 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750760    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="offline"
May 28 08:27:33.751398 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750769    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="offline"
May 28 08:27:33.751398 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750780    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="registry"
May 28 08:27:33.751398 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750790    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="online"
May 28 08:27:33.751398 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750800    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="c7569e81-9e55-4dce-b346-69a7a400546f" containerName="ui"
May 28 08:27:33.751398 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.750813    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="f8fc5968-cb7a-4881-9279-028814014f1e" containerName="ui"
May 28 08:27:33.754759 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.754736    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:33.757350 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.757326    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\""
May 28 08:27:33.757484 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.757357    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\""
May 28 08:27:33.757484 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.757421    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\""
May 28 08:27:33.757609 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.757600    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\""
May 28 08:27:33.757609 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.757605    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\""
May 28 08:27:33.757721 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.757626    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-gwvkb\""
May 28 08:27:33.757921 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.757900    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\""
May 28 08:27:33.765540 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.765519    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"]
May 28 08:27:33.883005 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.882967    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:33.883179 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.883012    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:33.883179 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.883039    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg44j\" (UniqueName: \"kubernetes.io/projected/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-kube-api-access-gg44j\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:33.883179 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.883163    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:33.883320 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.883220    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:33.883320 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.883249    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:33.984123 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.984092    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:33.984298 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.984135    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:33.984298 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.984156    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:33.984298 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.984214    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:33.984298 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.984247    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:33.984298 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.984271    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gg44j\" (UniqueName: \"kubernetes.io/projected/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-kube-api-access-gg44j\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:33.984597 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.984575    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:33.984678 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:27:33.984640    2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 28 08:27:33.984735 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:27:33.984674    2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 28 08:27:33.984735 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:27:33.984693    2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found
May 28 08:27:33.984735 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:27:33.984708    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-online-tls podName:4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e nodeName:}" failed. No retries permitted until 2026-05-28 08:27:34.484687636 +0000 UTC m=+1706.924759922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-online-tls") pod "feast-simple-feast-setup-664fcfcc58-gsfmh" (UID: "4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e") : secret "feast-simple-feast-setup-online-tls" not found
May 28 08:27:33.984896 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:27:33.984738    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-registry-tls podName:4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e nodeName:}" failed. No retries permitted until 2026-05-28 08:27:34.484718372 +0000 UTC m=+1706.924790643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-registry-tls") pod "feast-simple-feast-setup-664fcfcc58-gsfmh" (UID: "4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e") : secret "feast-simple-feast-setup-registry-tls" not found
May 28 08:27:33.984896 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:27:33.984754    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-offline-tls podName:4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e nodeName:}" failed. No retries permitted until 2026-05-28 08:27:34.484745757 +0000 UTC m=+1706.924818030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-offline-tls") pod "feast-simple-feast-setup-664fcfcc58-gsfmh" (UID: "4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e") : secret "feast-simple-feast-setup-offline-tls" not found
May 28 08:27:33.986699 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.986681    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:33.992924 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:33.992902    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg44j\" (UniqueName: \"kubernetes.io/projected/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-kube-api-access-gg44j\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:34.489146 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:34.489111    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:34.489305 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:34.489178    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:34.489305 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:34.489225    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:34.491571 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:34.491539    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:34.491710 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:34.491610    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:34.491710 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:34.491609    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-gsfmh\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:34.672614 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:34.672581    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:34.806236 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:34.806203    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"]
May 28 08:27:34.807560 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:27:34.807538    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fe66ef5_c15c_44dc_bf1e_bdd447aeef9e.slice/crio-08682b81eac173a4bc7ee6e5653e46e0d191868e5f34b74e9efeb9a47ba8a4a4 WatchSource:0}: Error finding container 08682b81eac173a4bc7ee6e5653e46e0d191868e5f34b74e9efeb9a47ba8a4a4: Status 404 returned error can't find the container with id 08682b81eac173a4bc7ee6e5653e46e0d191868e5f34b74e9efeb9a47ba8a4a4
May 28 08:27:35.553980 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:35.553946    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" event={"ID":"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e","Type":"ContainerStarted","Data":"e586b856b5e1bc5527405a0aa3e06ab8537593a86c8598282f041dba7713cf96"}
May 28 08:27:35.553980 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:35.553981    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" event={"ID":"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e","Type":"ContainerStarted","Data":"08682b81eac173a4bc7ee6e5653e46e0d191868e5f34b74e9efeb9a47ba8a4a4"}
May 28 08:27:39.566481 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:39.566441    2551 generic.go:358] "Generic (PLEG): container finished" podID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerID="e586b856b5e1bc5527405a0aa3e06ab8537593a86c8598282f041dba7713cf96" exitCode=0
May 28 08:27:39.566976 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:39.566509    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" event={"ID":"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e","Type":"ContainerDied","Data":"e586b856b5e1bc5527405a0aa3e06ab8537593a86c8598282f041dba7713cf96"}
May 28 08:27:40.571241 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:40.571205    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" event={"ID":"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e","Type":"ContainerStarted","Data":"d1742a300d942717d70001c01ce3bbade74f86b5daa3f7069619eb6bd13fb7e9"}
May 28 08:27:45.587234 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:45.587198    2551 generic.go:358] "Generic (PLEG): container finished" podID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerID="d1742a300d942717d70001c01ce3bbade74f86b5daa3f7069619eb6bd13fb7e9" exitCode=0
May 28 08:27:45.587676 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:45.587249    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" event={"ID":"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e","Type":"ContainerDied","Data":"d1742a300d942717d70001c01ce3bbade74f86b5daa3f7069619eb6bd13fb7e9"}
May 28 08:27:46.595180 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:46.595137    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" event={"ID":"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e","Type":"ContainerStarted","Data":"09214097aecd39c354621232e03b2e1bfa9e1bb8d304e0d5c4b526053b700b2a"}
May 28 08:27:46.595675 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:46.595188    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" event={"ID":"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e","Type":"ContainerStarted","Data":"a8e7186b9586535f2dbe1c61a801667b12c8166c299ff51934e5e1c6afdef63f"}
May 28 08:27:46.595675 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:46.595204    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" event={"ID":"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e","Type":"ContainerStarted","Data":"b8159d2de8371b1b8b7e5dfa2e40f0a569dab773b63b690f51bd3557fe1d6080"}
May 28 08:27:46.595675 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:46.595217    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" event={"ID":"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e","Type":"ContainerStarted","Data":"9aa57e0c36fb772c649f3f8f94411335587a9b65c9d4c7e175b57b6c840c49bf"}
May 28 08:27:46.620898 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:46.620828    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podStartSLOduration=13.620808061 podStartE2EDuration="13.620808061s" podCreationTimestamp="2026-05-28 08:27:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:27:46.618925284 +0000 UTC m=+1719.058997574" watchObservedRunningTime="2026-05-28 08:27:46.620808061 +0000 UTC m=+1719.060880351"
May 28 08:27:46.673111 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:46.673009    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:46.673111 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:46.673072    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:46.673111 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:46.673087    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:46.673571 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:46.673544    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:46.675615 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:46.675469    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.34:8816: connect: connection refused"
May 28 08:27:46.675615 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:46.675497    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused"
May 28 08:27:46.675615 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:46.675541    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused"
May 28 08:27:46.675841 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:46.675613    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.34:6571: connect: connection refused"
May 28 08:27:49.673511 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:49.673467    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.34:8816: connect: connection refused"
May 28 08:27:49.674090 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:49.673482    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused"
May 28 08:27:49.674209 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:49.673493    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.34:6571: connect: connection refused"
May 28 08:27:49.674209 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:49.673498    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused"
May 28 08:27:52.673688 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:52.673651    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:52.674099 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:52.673846    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:52.674254 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:52.674229    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:52.674338 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:52.674264    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:52.674338 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:52.674278    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:52.674338 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:52.674289    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:52.674525 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:52.674515    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:52.674652 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:52.674634    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:52.674761 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:52.674745    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:52.678147 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:52.678130    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:53.619654 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:53.619625    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:27:53.622655 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:27:53.622636    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:28:55.793700 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:55.793666    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"]
May 28 08:28:55.794153 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:55.793997    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="registry" containerID="cri-o://9aa57e0c36fb772c649f3f8f94411335587a9b65c9d4c7e175b57b6c840c49bf" gracePeriod=30
May 28 08:28:55.794153 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:55.794081    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="ui" containerID="cri-o://09214097aecd39c354621232e03b2e1bfa9e1bb8d304e0d5c4b526053b700b2a" gracePeriod=30
May 28 08:28:55.794153 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:55.794134    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="online" containerID="cri-o://b8159d2de8371b1b8b7e5dfa2e40f0a569dab773b63b690f51bd3557fe1d6080" gracePeriod=30
May 28 08:28:55.794534 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:55.794091    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="offline" containerID="cri-o://a8e7186b9586535f2dbe1c61a801667b12c8166c299ff51934e5e1c6afdef63f" gracePeriod=30
May 28 08:28:56.083155 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.083059    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"]
May 28 08:28:56.086950 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.086928    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.089606 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.089578    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-28c7f\""
May 28 08:28:56.101178 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.101126    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"]
May 28 08:28:56.186450 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.186405    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.186660 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.186466    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.186660 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.186497    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7tbb\" (UniqueName: \"kubernetes.io/projected/3eb81d49-4db2-4ab9-b75f-01dd473a0389-kube-api-access-l7tbb\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.186660 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.186546    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb81d49-4db2-4ab9-b75f-01dd473a0389-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.186827 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.186660    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.186827 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.186711    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.287615 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.287560    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb81d49-4db2-4ab9-b75f-01dd473a0389-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.287806 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.287666    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.287806 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.287694    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.287806 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.287742    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.287806 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.287769    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.287806 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.287796    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l7tbb\" (UniqueName: \"kubernetes.io/projected/3eb81d49-4db2-4ab9-b75f-01dd473a0389-kube-api-access-l7tbb\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.288085 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:28:56.287834    2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found
May 28 08:28:56.288085 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:28:56.287914    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-ui-tls podName:3eb81d49-4db2-4ab9-b75f-01dd473a0389 nodeName:}" failed. No retries permitted until 2026-05-28 08:28:56.787892604 +0000 UTC m=+1789.227964887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-ui-tls") pod "feast-simple-feast-setup-664fcfcc58-5d7xz" (UID: "3eb81d49-4db2-4ab9-b75f-01dd473a0389") : secret "feast-simple-feast-setup-ui-tls" not found
May 28 08:28:56.288085 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:28:56.288081    2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found
May 28 08:28:56.288247 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:28:56.288119    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-registry-tls podName:3eb81d49-4db2-4ab9-b75f-01dd473a0389 nodeName:}" failed. No retries permitted until 2026-05-28 08:28:56.788107119 +0000 UTC m=+1789.228179401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-registry-tls") pod "feast-simple-feast-setup-664fcfcc58-5d7xz" (UID: "3eb81d49-4db2-4ab9-b75f-01dd473a0389") : secret "feast-simple-feast-setup-registry-tls" not found
May 28 08:28:56.288247 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:28:56.288205    2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found
May 28 08:28:56.288338 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:28:56.288250    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-online-tls podName:3eb81d49-4db2-4ab9-b75f-01dd473a0389 nodeName:}" failed. No retries permitted until 2026-05-28 08:28:56.788238814 +0000 UTC m=+1789.228311099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-online-tls") pod "feast-simple-feast-setup-664fcfcc58-5d7xz" (UID: "3eb81d49-4db2-4ab9-b75f-01dd473a0389") : secret "feast-simple-feast-setup-online-tls" not found
May 28 08:28:56.288613 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.288591    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb81d49-4db2-4ab9-b75f-01dd473a0389-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.290878 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.290846    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.299358 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.299331    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7tbb\" (UniqueName: \"kubernetes.io/projected/3eb81d49-4db2-4ab9-b75f-01dd473a0389-kube-api-access-l7tbb\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.793274 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.793237    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.793471 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.793325    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.793471 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.793434    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.795812 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.795784    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.795812 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.795811    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.796148 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.795839    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-5d7xz\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:56.810340 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.810310    2551 generic.go:358] "Generic (PLEG): container finished" podID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerID="09214097aecd39c354621232e03b2e1bfa9e1bb8d304e0d5c4b526053b700b2a" exitCode=0
May 28 08:28:56.810340 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.810335    2551 generic.go:358] "Generic (PLEG): container finished" podID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerID="b8159d2de8371b1b8b7e5dfa2e40f0a569dab773b63b690f51bd3557fe1d6080" exitCode=0
May 28 08:28:56.810513 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.810401    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" event={"ID":"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e","Type":"ContainerDied","Data":"09214097aecd39c354621232e03b2e1bfa9e1bb8d304e0d5c4b526053b700b2a"}
May 28 08:28:56.810513 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:56.810434    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" event={"ID":"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e","Type":"ContainerDied","Data":"b8159d2de8371b1b8b7e5dfa2e40f0a569dab773b63b690f51bd3557fe1d6080"}
May 28 08:28:57.003045 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:57.003010    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:28:57.134583 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:57.134554    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"]
May 28 08:28:57.136165 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:28:57.136140    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eb81d49_4db2_4ab9_b75f_01dd473a0389.slice/crio-c843275f43fe2b3a3c4931db7b1f151079d0c5be423006ac05657126718fd6b0 WatchSource:0}: Error finding container c843275f43fe2b3a3c4931db7b1f151079d0c5be423006ac05657126718fd6b0: Status 404 returned error can't find the container with id c843275f43fe2b3a3c4931db7b1f151079d0c5be423006ac05657126718fd6b0
May 28 08:28:57.814996 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:57.814959    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" event={"ID":"3eb81d49-4db2-4ab9-b75f-01dd473a0389","Type":"ContainerStarted","Data":"f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427"}
May 28 08:28:57.814996 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:28:57.814999    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" event={"ID":"3eb81d49-4db2-4ab9-b75f-01dd473a0389","Type":"ContainerStarted","Data":"c843275f43fe2b3a3c4931db7b1f151079d0c5be423006ac05657126718fd6b0"}
May 28 08:29:01.827437 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:01.827403    2551 generic.go:358] "Generic (PLEG): container finished" podID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerID="f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427" exitCode=0
May 28 08:29:01.827797 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:01.827473    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" event={"ID":"3eb81d49-4db2-4ab9-b75f-01dd473a0389","Type":"ContainerDied","Data":"f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427"}
May 28 08:29:02.675023 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:02.674971    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused"
May 28 08:29:02.832563 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:02.832529    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" event={"ID":"3eb81d49-4db2-4ab9-b75f-01dd473a0389","Type":"ContainerStarted","Data":"1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118"}
May 28 08:29:03.620682 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:03.620643    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused"
May 28 08:29:07.848034 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:07.848003    2551 generic.go:358] "Generic (PLEG): container finished" podID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerID="1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118" exitCode=0
May 28 08:29:07.848436 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:07.848086    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" event={"ID":"3eb81d49-4db2-4ab9-b75f-01dd473a0389","Type":"ContainerDied","Data":"1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118"}
May 28 08:29:08.281682 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:08.281655    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:29:08.282224 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:08.282078    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:29:08.856069 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:08.856028    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" event={"ID":"3eb81d49-4db2-4ab9-b75f-01dd473a0389","Type":"ContainerStarted","Data":"1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef"}
May 28 08:29:08.856069 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:08.856074    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" event={"ID":"3eb81d49-4db2-4ab9-b75f-01dd473a0389","Type":"ContainerStarted","Data":"215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c"}
May 28 08:29:08.856626 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:08.856091    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" event={"ID":"3eb81d49-4db2-4ab9-b75f-01dd473a0389","Type":"ContainerStarted","Data":"ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46"}
May 28 08:29:08.856626 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:08.856103    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" event={"ID":"3eb81d49-4db2-4ab9-b75f-01dd473a0389","Type":"ContainerStarted","Data":"016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997"}
May 28 08:29:08.882208 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:08.882141    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podStartSLOduration=12.882122394 podStartE2EDuration="12.882122394s" podCreationTimestamp="2026-05-28 08:28:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:29:08.87929768 +0000 UTC m=+1801.319369982" watchObservedRunningTime="2026-05-28 08:29:08.882122394 +0000 UTC m=+1801.322194684"
May 28 08:29:09.004142 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:09.004074    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:09.004142 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:09.004121    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:09.004142 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:09.004136    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:09.004142 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:09.004149    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:09.006379 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:09.006344    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused"
May 28 08:29:09.006489 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:09.006364    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.35:8816: connect: connection refused"
May 28 08:29:09.006548 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:09.006473    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 28 08:29:09.006631 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:09.006361    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.35:6571: connect: connection refused"
May 28 08:29:12.004101 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:12.004056    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.35:6571: connect: connection refused"
May 28 08:29:12.004704 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:12.004056    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused"
May 28 08:29:12.004840 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:12.004084    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.35:8816: connect: connection refused"
May 28 08:29:12.004840 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:12.004159    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 28 08:29:12.675616 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:12.675242    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused"
May 28 08:29:13.620593 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:13.620552    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused"
May 28 08:29:15.004821 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:15.004782    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:15.005213 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:15.004928    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:15.005272 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:15.005252    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:15.005631 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:15.005603    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:15.005750 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:15.005638    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:15.005750 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:15.005650    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:15.005750 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:15.005694    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:15.005916 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:15.005785    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:15.005916 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:15.005903    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:15.009392 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:15.009361    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:15.880239 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:15.880207    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:15.883164 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:15.883147    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:29:22.674740 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:22.674694    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused"
May 28 08:29:22.675218 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:22.674855    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:29:23.619996 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:23.619947    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused"
May 28 08:29:23.620182 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:23.620078    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:29:25.914688 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:25.914656    2551 generic.go:358] "Generic (PLEG): container finished" podID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerID="a8e7186b9586535f2dbe1c61a801667b12c8166c299ff51934e5e1c6afdef63f" exitCode=137
May 28 08:29:25.914688 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:25.914686    2551 generic.go:358] "Generic (PLEG): container finished" podID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerID="9aa57e0c36fb772c649f3f8f94411335587a9b65c9d4c7e175b57b6c840c49bf" exitCode=137
May 28 08:29:25.915101 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:25.914714    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" event={"ID":"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e","Type":"ContainerDied","Data":"a8e7186b9586535f2dbe1c61a801667b12c8166c299ff51934e5e1c6afdef63f"}
May 28 08:29:25.915101 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:25.914741    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" event={"ID":"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e","Type":"ContainerDied","Data":"9aa57e0c36fb772c649f3f8f94411335587a9b65c9d4c7e175b57b6c840c49bf"}
May 28 08:29:26.448441 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.448417    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:29:26.569068 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.568977    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-ui-tls\") pod \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") "
May 28 08:29:26.569238 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.569071    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-online-tls\") pod \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") "
May 28 08:29:26.569238 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.569110    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg44j\" (UniqueName: \"kubernetes.io/projected/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-kube-api-access-gg44j\") pod \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") "
May 28 08:29:26.569238 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.569159    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-registry-tls\") pod \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") "
May 28 08:29:26.569238 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.569196    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-feast-data\") pod \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") "
May 28 08:29:26.569238 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.569217    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-offline-tls\") pod \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\" (UID: \"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e\") "
May 28 08:29:26.569816 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.569784    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-feast-data" (OuterVolumeSpecName: "feast-data") pod "4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" (UID: "4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 08:29:26.571560 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.571527    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" (UID: "4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:29:26.571560 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.571544    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-kube-api-access-gg44j" (OuterVolumeSpecName: "kube-api-access-gg44j") pod "4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" (UID: "4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e"). InnerVolumeSpecName "kube-api-access-gg44j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:29:26.571739 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.571598    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" (UID: "4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:29:26.571739 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.571677    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-online-tls" (OuterVolumeSpecName: "online-tls") pod "4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" (UID: "4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:29:26.571739 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.571713    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" (UID: "4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:29:26.670110 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.670078    2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-feast-data\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:29:26.670110 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.670105    2551 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-offline-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:29:26.670110 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.670115    2551 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-ui-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:29:26.670326 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.670123    2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-online-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:29:26.670326 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.670133    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg44j\" (UniqueName: \"kubernetes.io/projected/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-kube-api-access-gg44j\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:29:26.670326 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.670142    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e-registry-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:29:26.920447 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.920344    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh" event={"ID":"4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e","Type":"ContainerDied","Data":"08682b81eac173a4bc7ee6e5653e46e0d191868e5f34b74e9efeb9a47ba8a4a4"}
May 28 08:29:26.920447 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.920426    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"
May 28 08:29:26.920853 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.920428    2551 scope.go:117] "RemoveContainer" containerID="09214097aecd39c354621232e03b2e1bfa9e1bb8d304e0d5c4b526053b700b2a"
May 28 08:29:26.929892 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.929871    2551 scope.go:117] "RemoveContainer" containerID="a8e7186b9586535f2dbe1c61a801667b12c8166c299ff51934e5e1c6afdef63f"
May 28 08:29:26.936885 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.936861    2551 scope.go:117] "RemoveContainer" containerID="b8159d2de8371b1b8b7e5dfa2e40f0a569dab773b63b690f51bd3557fe1d6080"
May 28 08:29:26.942028 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.942004    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"]
May 28 08:29:26.944218 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.944198    2551 scope.go:117] "RemoveContainer" containerID="9aa57e0c36fb772c649f3f8f94411335587a9b65c9d4c7e175b57b6c840c49bf"
May 28 08:29:26.945596 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.945577    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-gsfmh"]
May 28 08:29:26.951449 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.951431    2551 scope.go:117] "RemoveContainer" containerID="d1742a300d942717d70001c01ce3bbade74f86b5daa3f7069619eb6bd13fb7e9"
May 28 08:29:26.958181 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:26.958160    2551 scope.go:117] "RemoveContainer" containerID="e586b856b5e1bc5527405a0aa3e06ab8537593a86c8598282f041dba7713cf96"
May 28 08:29:28.194894 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:29:28.194860    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" path="/var/lib/kubelet/pods/4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e/volumes"
May 28 08:30:17.576711 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.576676    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"]
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.576988    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="feast-apply"
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.576998    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="feast-apply"
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.577006    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="registry"
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.577011    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="registry"
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.577019    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="ui"
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.577025    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="ui"
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.577034    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="online"
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.577038    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="online"
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.577049    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="feast-init"
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.577054    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="feast-init"
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.577060    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="offline"
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.577065    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="offline"
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.577125    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="offline"
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.577134    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="ui"
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.577142    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="registry"
May 28 08:30:17.579444 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.577150    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="4fe66ef5-c15c-44dc-bf1e-bdd447aeef9e" containerName="online"
May 28 08:30:17.580300 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.580284    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.582896 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.582871    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\""
May 28 08:30:17.582999 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.582873    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-zt46d\""
May 28 08:30:17.582999 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.582871    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\""
May 28 08:30:17.582999 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.582881    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\""
May 28 08:30:17.583851 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.583834    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\""
May 28 08:30:17.583912 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.583893    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\""
May 28 08:30:17.583999 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.583983    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\""
May 28 08:30:17.589012 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.588987    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"]
May 28 08:30:17.610977 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.610939    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.610977 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.610985    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.611213 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.611017    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0c202a7e-f36e-4467-8473-ffc852dc6f0d-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.611213 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.611043    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtzmt\" (UniqueName: \"kubernetes.io/projected/0c202a7e-f36e-4467-8473-ffc852dc6f0d-kube-api-access-xtzmt\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.611213 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.611184    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0c202a7e-f36e-4467-8473-ffc852dc6f0d-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.611350 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.611234    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.712485 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.712443    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0c202a7e-f36e-4467-8473-ffc852dc6f0d-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.712485 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.712492    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.712710 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.712543    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.712710 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.712559    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.712710 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.712581    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0c202a7e-f36e-4467-8473-ffc852dc6f0d-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.712710 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.712607    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xtzmt\" (UniqueName: \"kubernetes.io/projected/0c202a7e-f36e-4467-8473-ffc852dc6f0d-kube-api-access-xtzmt\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.713005 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.712982    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0c202a7e-f36e-4467-8473-ffc852dc6f0d-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.713220 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.713201    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0c202a7e-f36e-4467-8473-ffc852dc6f0d-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.715232 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.715210    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.715232 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.715231    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.715371 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.715265    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.720336 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.720313    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtzmt\" (UniqueName: \"kubernetes.io/projected/0c202a7e-f36e-4467-8473-ffc852dc6f0d-kube-api-access-xtzmt\") pod \"feast-simple-feast-remote-setup-5cff94746b-npx4b\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:17.890400 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:17.890288    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:18.019871 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:18.019839    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"]
May 28 08:30:18.021722 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:30:18.021695    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c202a7e_f36e_4467_8473_ffc852dc6f0d.slice/crio-5a5a9615d92a870861f07027ecdb42183a6942aa426ce934322ef0c7394ddcb4 WatchSource:0}: Error finding container 5a5a9615d92a870861f07027ecdb42183a6942aa426ce934322ef0c7394ddcb4: Status 404 returned error can't find the container with id 5a5a9615d92a870861f07027ecdb42183a6942aa426ce934322ef0c7394ddcb4
May 28 08:30:18.023466 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:18.023449    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 28 08:30:18.076500 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:18.076467    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" event={"ID":"0c202a7e-f36e-4467-8473-ffc852dc6f0d","Type":"ContainerStarted","Data":"5a5a9615d92a870861f07027ecdb42183a6942aa426ce934322ef0c7394ddcb4"}
May 28 08:30:19.081605 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:19.081521    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" event={"ID":"0c202a7e-f36e-4467-8473-ffc852dc6f0d","Type":"ContainerStarted","Data":"53a6f59e1a40a09c41ae714cdabe5fd0197b284470960a84257935f56516e82c"}
May 28 08:30:23.095288 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:23.095251    2551 generic.go:358] "Generic (PLEG): container finished" podID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerID="53a6f59e1a40a09c41ae714cdabe5fd0197b284470960a84257935f56516e82c" exitCode=0
May 28 08:30:23.095667 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:23.095315    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" event={"ID":"0c202a7e-f36e-4467-8473-ffc852dc6f0d","Type":"ContainerDied","Data":"53a6f59e1a40a09c41ae714cdabe5fd0197b284470960a84257935f56516e82c"}
May 28 08:30:24.100164 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:24.100129    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" event={"ID":"0c202a7e-f36e-4467-8473-ffc852dc6f0d","Type":"ContainerStarted","Data":"e94bda03810502285373a8c5fbda1cac5ee43ff6d9613f3e4b0c4c250ea23f88"}
May 28 08:30:29.116773 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:29.116684    2551 generic.go:358] "Generic (PLEG): container finished" podID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerID="e94bda03810502285373a8c5fbda1cac5ee43ff6d9613f3e4b0c4c250ea23f88" exitCode=0
May 28 08:30:29.116773 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:29.116736    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" event={"ID":"0c202a7e-f36e-4467-8473-ffc852dc6f0d","Type":"ContainerDied","Data":"e94bda03810502285373a8c5fbda1cac5ee43ff6d9613f3e4b0c4c250ea23f88"}
May 28 08:30:30.124470 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:30.124420    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" event={"ID":"0c202a7e-f36e-4467-8473-ffc852dc6f0d","Type":"ContainerStarted","Data":"1496f964a80267aabfd2c6b6e4b7c233572829566d59fb2b98af3f36d05f29e4"}
May 28 08:30:30.124921 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:30.124481    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" event={"ID":"0c202a7e-f36e-4467-8473-ffc852dc6f0d","Type":"ContainerStarted","Data":"6e5f74435c0c31ba37422900b7a6b2d7ffa81e3c0f51e0df691f915514fb82e7"}
May 28 08:30:30.124921 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:30.124498    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" event={"ID":"0c202a7e-f36e-4467-8473-ffc852dc6f0d","Type":"ContainerStarted","Data":"5ebe51e5633ff389a26803a249f1ff04f33db4de224e56a036e322e774bc803c"}
May 28 08:30:30.148431 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:30.148362    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" podStartSLOduration=13.148345876 podStartE2EDuration="13.148345876s" podCreationTimestamp="2026-05-28 08:30:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:30:30.146007836 +0000 UTC m=+1882.586080139" watchObservedRunningTime="2026-05-28 08:30:30.148345876 +0000 UTC m=+1882.588418165"
May 28 08:30:32.891010 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:32.890963    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:32.891010 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:32.891022    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:32.891567 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:32.891038    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:32.892500 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:32.892464    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused"
May 28 08:30:32.892596 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:32.892470    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.36:8816: connect: connection refused"
May 28 08:30:32.892659 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:32.892471    2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused"
May 28 08:30:35.892458 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:35.892427    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:35.892865 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:35.892525    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:35.892946 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:35.892882    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:35.892946 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:35.892917    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:35.893290 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:35.893269    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:35.893363 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:35.893347    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:35.896189 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:35.896173    2551 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:36.148296 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:36.148200    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:30:36.151196 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:30:36.151179    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:31:38.577569 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:38.577533    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"]
May 28 08:31:38.578115 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:38.577990    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="online" containerID="cri-o://5ebe51e5633ff389a26803a249f1ff04f33db4de224e56a036e322e774bc803c" gracePeriod=30
May 28 08:31:38.578196 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:38.578089    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="ui" containerID="cri-o://1496f964a80267aabfd2c6b6e4b7c233572829566d59fb2b98af3f36d05f29e4" gracePeriod=30
May 28 08:31:38.578196 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:38.578133    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="offline" containerID="cri-o://6e5f74435c0c31ba37422900b7a6b2d7ffa81e3c0f51e0df691f915514fb82e7" gracePeriod=30
May 28 08:31:38.720571 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:38.720538    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"]
May 28 08:31:38.721079 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:38.720882    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="registry" containerID="cri-o://016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997" gracePeriod=30
May 28 08:31:38.721079 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:38.720914    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="offline" containerID="cri-o://215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c" gracePeriod=30
May 28 08:31:38.721079 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:38.720936    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="online" containerID="cri-o://ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46" gracePeriod=30
May 28 08:31:38.721079 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:38.720960    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="ui" containerID="cri-o://1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef" gracePeriod=30
May 28 08:31:39.332355 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:39.332271    2551 generic.go:358] "Generic (PLEG): container finished" podID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerID="1496f964a80267aabfd2c6b6e4b7c233572829566d59fb2b98af3f36d05f29e4" exitCode=0
May 28 08:31:39.332355 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:39.332299    2551 generic.go:358] "Generic (PLEG): container finished" podID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerID="5ebe51e5633ff389a26803a249f1ff04f33db4de224e56a036e322e774bc803c" exitCode=0
May 28 08:31:39.332355 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:39.332341    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" event={"ID":"0c202a7e-f36e-4467-8473-ffc852dc6f0d","Type":"ContainerDied","Data":"1496f964a80267aabfd2c6b6e4b7c233572829566d59fb2b98af3f36d05f29e4"}
May 28 08:31:39.332636 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:39.332401    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" event={"ID":"0c202a7e-f36e-4467-8473-ffc852dc6f0d","Type":"ContainerDied","Data":"5ebe51e5633ff389a26803a249f1ff04f33db4de224e56a036e322e774bc803c"}
May 28 08:31:39.334834 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:39.334811    2551 generic.go:358] "Generic (PLEG): container finished" podID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerID="1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef" exitCode=0
May 28 08:31:39.334937 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:39.334896    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" event={"ID":"3eb81d49-4db2-4ab9-b75f-01dd473a0389","Type":"ContainerDied","Data":"1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef"}
May 28 08:31:40.340157 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:40.340124    2551 generic.go:358] "Generic (PLEG): container finished" podID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerID="ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46" exitCode=0
May 28 08:31:40.340528 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:40.340191    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" event={"ID":"3eb81d49-4db2-4ab9-b75f-01dd473a0389","Type":"ContainerDied","Data":"ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46"}
May 28 08:31:45.006560 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:45.006495    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused"
May 28 08:31:45.880644 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:45.880595    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 28 08:31:45.893918 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:45.893881    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused"
May 28 08:31:46.149447 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:46.149352    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused"
May 28 08:31:55.006570 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:55.006464    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused"
May 28 08:31:55.880453 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:55.880378    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 28 08:31:55.893107 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:55.893068    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused"
May 28 08:31:56.148812 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:31:56.148717    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused"
May 28 08:32:05.006103 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:05.006055    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused"
May 28 08:32:05.006518 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:05.006229    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:32:05.881144 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:05.881097    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused"
May 28 08:32:05.881303 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:05.881220    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:32:05.893878 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:05.893833    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused"
May 28 08:32:05.894015 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:05.893950    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:32:06.149419 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:06.149315    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused"
May 28 08:32:06.149766 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:06.149473    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:32:08.833512 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:08.833444    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:32:08.951187 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:08.951138    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-offline-tls\") pod \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") "
May 28 08:32:08.951187 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:08.951195    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0c202a7e-f36e-4467-8473-ffc852dc6f0d-feast-data\") pod \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") "
May 28 08:32:08.951479 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:08.951237    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-ui-tls\") pod \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") "
May 28 08:32:08.951479 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:08.951320    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-online-tls\") pod \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") "
May 28 08:32:08.951479 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:08.951370    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtzmt\" (UniqueName: \"kubernetes.io/projected/0c202a7e-f36e-4467-8473-ffc852dc6f0d-kube-api-access-xtzmt\") pod \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") "
May 28 08:32:08.951479 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:08.951457    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0c202a7e-f36e-4467-8473-ffc852dc6f0d-registry-tls\") pod \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\" (UID: \"0c202a7e-f36e-4467-8473-ffc852dc6f0d\") "
May 28 08:32:08.951842 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:08.951795    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c202a7e-f36e-4467-8473-ffc852dc6f0d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0c202a7e-f36e-4467-8473-ffc852dc6f0d" (UID: "0c202a7e-f36e-4467-8473-ffc852dc6f0d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 28 08:32:08.951970 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:08.951845    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c202a7e-f36e-4467-8473-ffc852dc6f0d-feast-data" (OuterVolumeSpecName: "feast-data") pod "0c202a7e-f36e-4467-8473-ffc852dc6f0d" (UID: "0c202a7e-f36e-4467-8473-ffc852dc6f0d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 08:32:08.953596 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:08.953565    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "0c202a7e-f36e-4467-8473-ffc852dc6f0d" (UID: "0c202a7e-f36e-4467-8473-ffc852dc6f0d"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:32:08.953596 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:08.953580    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-online-tls" (OuterVolumeSpecName: "online-tls") pod "0c202a7e-f36e-4467-8473-ffc852dc6f0d" (UID: "0c202a7e-f36e-4467-8473-ffc852dc6f0d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:32:08.953763 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:08.953741    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "0c202a7e-f36e-4467-8473-ffc852dc6f0d" (UID: "0c202a7e-f36e-4467-8473-ffc852dc6f0d"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:32:08.953801 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:08.953750    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c202a7e-f36e-4467-8473-ffc852dc6f0d-kube-api-access-xtzmt" (OuterVolumeSpecName: "kube-api-access-xtzmt") pod "0c202a7e-f36e-4467-8473-ffc852dc6f0d" (UID: "0c202a7e-f36e-4467-8473-ffc852dc6f0d"). InnerVolumeSpecName "kube-api-access-xtzmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:32:09.052734 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.052702    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtzmt\" (UniqueName: \"kubernetes.io/projected/0c202a7e-f36e-4467-8473-ffc852dc6f0d-kube-api-access-xtzmt\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:32:09.052734 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.052730    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/0c202a7e-f36e-4467-8473-ffc852dc6f0d-registry-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:32:09.052734 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.052741    2551 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-offline-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:32:09.052955 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.052749    2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0c202a7e-f36e-4467-8473-ffc852dc6f0d-feast-data\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:32:09.052955 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.052757    2551 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-ui-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:32:09.052955 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.052764    2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0c202a7e-f36e-4467-8473-ffc852dc6f0d-online-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:32:09.363883 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.363858    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:32:09.426093 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.426051    2551 generic.go:358] "Generic (PLEG): container finished" podID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerID="215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c" exitCode=137
May 28 08:32:09.426093 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.426082    2551 generic.go:358] "Generic (PLEG): container finished" podID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerID="016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997" exitCode=137
May 28 08:32:09.426351 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.426139    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"
May 28 08:32:09.426351 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.426141    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" event={"ID":"3eb81d49-4db2-4ab9-b75f-01dd473a0389","Type":"ContainerDied","Data":"215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c"}
May 28 08:32:09.426351 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.426181    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" event={"ID":"3eb81d49-4db2-4ab9-b75f-01dd473a0389","Type":"ContainerDied","Data":"016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997"}
May 28 08:32:09.426351 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.426197    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz" event={"ID":"3eb81d49-4db2-4ab9-b75f-01dd473a0389","Type":"ContainerDied","Data":"c843275f43fe2b3a3c4931db7b1f151079d0c5be423006ac05657126718fd6b0"}
May 28 08:32:09.426351 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.426218    2551 scope.go:117] "RemoveContainer" containerID="1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef"
May 28 08:32:09.428349 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.428318    2551 generic.go:358] "Generic (PLEG): container finished" podID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerID="6e5f74435c0c31ba37422900b7a6b2d7ffa81e3c0f51e0df691f915514fb82e7" exitCode=137
May 28 08:32:09.428487 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.428358    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" event={"ID":"0c202a7e-f36e-4467-8473-ffc852dc6f0d","Type":"ContainerDied","Data":"6e5f74435c0c31ba37422900b7a6b2d7ffa81e3c0f51e0df691f915514fb82e7"}
May 28 08:32:09.428487 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.428393    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"
May 28 08:32:09.428487 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.428400    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b" event={"ID":"0c202a7e-f36e-4467-8473-ffc852dc6f0d","Type":"ContainerDied","Data":"5a5a9615d92a870861f07027ecdb42183a6942aa426ce934322ef0c7394ddcb4"}
May 28 08:32:09.441051 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.441035    2551 scope.go:117] "RemoveContainer" containerID="215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c"
May 28 08:32:09.448747 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.448726    2551 scope.go:117] "RemoveContainer" containerID="ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46"
May 28 08:32:09.456021 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.456006    2551 scope.go:117] "RemoveContainer" containerID="016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997"
May 28 08:32:09.456096 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.456079    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7tbb\" (UniqueName: \"kubernetes.io/projected/3eb81d49-4db2-4ab9-b75f-01dd473a0389-kube-api-access-l7tbb\") pod \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") "
May 28 08:32:09.456150 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.456120    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-online-tls\") pod \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") "
May 28 08:32:09.456204 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.456189    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-registry-tls\") pod \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") "
May 28 08:32:09.456259 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.456252    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-ui-tls\") pod \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") "
May 28 08:32:09.456312 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.456287    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb81d49-4db2-4ab9-b75f-01dd473a0389-feast-data\") pod \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") "
May 28 08:32:09.456312 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.456310    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-offline-tls\") pod \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\" (UID: \"3eb81d49-4db2-4ab9-b75f-01dd473a0389\") "
May 28 08:32:09.457343 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.457310    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb81d49-4db2-4ab9-b75f-01dd473a0389-feast-data" (OuterVolumeSpecName: "feast-data") pod "3eb81d49-4db2-4ab9-b75f-01dd473a0389" (UID: "3eb81d49-4db2-4ab9-b75f-01dd473a0389"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 08:32:09.457581 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.457544    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"]
May 28 08:32:09.458625 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.458591    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-online-tls" (OuterVolumeSpecName: "online-tls") pod "3eb81d49-4db2-4ab9-b75f-01dd473a0389" (UID: "3eb81d49-4db2-4ab9-b75f-01dd473a0389"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:32:09.458774 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.458751    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "3eb81d49-4db2-4ab9-b75f-01dd473a0389" (UID: "3eb81d49-4db2-4ab9-b75f-01dd473a0389"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:32:09.458832 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.458769    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3eb81d49-4db2-4ab9-b75f-01dd473a0389" (UID: "3eb81d49-4db2-4ab9-b75f-01dd473a0389"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:32:09.458871 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.458829    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb81d49-4db2-4ab9-b75f-01dd473a0389-kube-api-access-l7tbb" (OuterVolumeSpecName: "kube-api-access-l7tbb") pod "3eb81d49-4db2-4ab9-b75f-01dd473a0389" (UID: "3eb81d49-4db2-4ab9-b75f-01dd473a0389"). InnerVolumeSpecName "kube-api-access-l7tbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:32:09.459080 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.459063    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "3eb81d49-4db2-4ab9-b75f-01dd473a0389" (UID: "3eb81d49-4db2-4ab9-b75f-01dd473a0389"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 28 08:32:09.462097 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.462077    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-npx4b"]
May 28 08:32:09.464937 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.464921    2551 scope.go:117] "RemoveContainer" containerID="1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118"
May 28 08:32:09.471858 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.471835    2551 scope.go:117] "RemoveContainer" containerID="f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427"
May 28 08:32:09.478186 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.478161    2551 scope.go:117] "RemoveContainer" containerID="1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef"
May 28 08:32:09.478454 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:32:09.478430    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef\": container with ID starting with 1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef not found: ID does not exist" containerID="1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef"
May 28 08:32:09.478523 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.478467    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef"} err="failed to get container status \"1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef\": rpc error: code = NotFound desc = could not find container \"1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef\": container with ID starting with 1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef not found: ID does not exist"
May 28 08:32:09.478523 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.478493    2551 scope.go:117] "RemoveContainer" containerID="215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c"
May 28 08:32:09.478756 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:32:09.478739    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c\": container with ID starting with 215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c not found: ID does not exist" containerID="215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c"
May 28 08:32:09.478800 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.478764    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c"} err="failed to get container status \"215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c\": rpc error: code = NotFound desc = could not find container \"215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c\": container with ID starting with 215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c not found: ID does not exist"
May 28 08:32:09.478800 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.478781    2551 scope.go:117] "RemoveContainer" containerID="ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46"
May 28 08:32:09.479018 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:32:09.478999    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46\": container with ID starting with ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46 not found: ID does not exist" containerID="ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46"
May 28 08:32:09.479068 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.479024    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46"} err="failed to get container status \"ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46\": rpc error: code = NotFound desc = could not find container \"ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46\": container with ID starting with ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46 not found: ID does not exist"
May 28 08:32:09.479068 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.479038    2551 scope.go:117] "RemoveContainer" containerID="016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997"
May 28 08:32:09.479255 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:32:09.479238    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997\": container with ID starting with 016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997 not found: ID does not exist" containerID="016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997"
May 28 08:32:09.479297 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.479259    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997"} err="failed to get container status \"016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997\": rpc error: code = NotFound desc = could not find container \"016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997\": container with ID starting with 016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997 not found: ID does not exist"
May 28 08:32:09.479297 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.479273    2551 scope.go:117] "RemoveContainer" containerID="1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118"
May 28 08:32:09.479510 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:32:09.479490    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118\": container with ID starting with 1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118 not found: ID does not exist" containerID="1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118"
May 28 08:32:09.479590 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.479513    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118"} err="failed to get container status \"1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118\": rpc error: code = NotFound desc = could not find container \"1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118\": container with ID starting with 1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118 not found: ID does not exist"
May 28 08:32:09.479590 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.479526    2551 scope.go:117] "RemoveContainer" containerID="f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427"
May 28 08:32:09.479728 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:32:09.479714    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427\": container with ID starting with f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427 not found: ID does not exist" containerID="f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427"
May 28 08:32:09.479764 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.479732    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427"} err="failed to get container status \"f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427\": rpc error: code = NotFound desc = could not find container \"f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427\": container with ID starting with f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427 not found: ID does not exist"
May 28 08:32:09.479764 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.479744    2551 scope.go:117] "RemoveContainer" containerID="1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef"
May 28 08:32:09.479953 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.479938    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef"} err="failed to get container status \"1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef\": rpc error: code = NotFound desc = could not find container \"1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef\": container with ID starting with 1759039858be765d23f7dc1501ea335914f3f0934ec6f3d43596eca8f2eca9ef not found: ID does not exist"
May 28 08:32:09.479998 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.479953    2551 scope.go:117] "RemoveContainer" containerID="215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c"
May 28 08:32:09.480119 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.480102    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c"} err="failed to get container status \"215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c\": rpc error: code = NotFound desc = could not find container \"215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c\": container with ID starting with 215d80eb2a5e21e9edfa6023fcde83c2496dc76e0f2a87e90d8181a22f82b93c not found: ID does not exist"
May 28 08:32:09.480165 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.480119    2551 scope.go:117] "RemoveContainer" containerID="ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46"
May 28 08:32:09.480270 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.480255    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46"} err="failed to get container status \"ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46\": rpc error: code = NotFound desc = could not find container \"ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46\": container with ID starting with ace9e792f7f39f2d4d9061e63608065436092bbc196c9763d45798bff3012f46 not found: ID does not exist"
May 28 08:32:09.480325 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.480270    2551 scope.go:117] "RemoveContainer" containerID="016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997"
May 28 08:32:09.480481 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.480467    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997"} err="failed to get container status \"016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997\": rpc error: code = NotFound desc = could not find container \"016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997\": container with ID starting with 016411ceffc6f311e1dc601975384610011b20020a67bc33d37822f9e78aa997 not found: ID does not exist"
May 28 08:32:09.480481 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.480481    2551 scope.go:117] "RemoveContainer" containerID="1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118"
May 28 08:32:09.480675 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.480660    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118"} err="failed to get container status \"1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118\": rpc error: code = NotFound desc = could not find container \"1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118\": container with ID starting with 1b44ca5638399d44d63bc640507f77ff933fbb6c20a651f280865eab3f983118 not found: ID does not exist"
May 28 08:32:09.480675 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.480674    2551 scope.go:117] "RemoveContainer" containerID="f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427"
May 28 08:32:09.480881 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.480863    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427"} err="failed to get container status \"f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427\": rpc error: code = NotFound desc = could not find container \"f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427\": container with ID starting with f313b2cca3a0e411ef6e2bb95958fe6b7c6bdcc6be91eda06b7f6fc9caf68427 not found: ID does not exist"
May 28 08:32:09.480923 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.480882    2551 scope.go:117] "RemoveContainer" containerID="1496f964a80267aabfd2c6b6e4b7c233572829566d59fb2b98af3f36d05f29e4"
May 28 08:32:09.487764 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.487755    2551 scope.go:117] "RemoveContainer" containerID="6e5f74435c0c31ba37422900b7a6b2d7ffa81e3c0f51e0df691f915514fb82e7"
May 28 08:32:09.494306 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.494292    2551 scope.go:117] "RemoveContainer" containerID="5ebe51e5633ff389a26803a249f1ff04f33db4de224e56a036e322e774bc803c"
May 28 08:32:09.500908 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.500893    2551 scope.go:117] "RemoveContainer" containerID="e94bda03810502285373a8c5fbda1cac5ee43ff6d9613f3e4b0c4c250ea23f88"
May 28 08:32:09.507442 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.507374    2551 scope.go:117] "RemoveContainer" containerID="53a6f59e1a40a09c41ae714cdabe5fd0197b284470960a84257935f56516e82c"
May 28 08:32:09.514086 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.514069    2551 scope.go:117] "RemoveContainer" containerID="1496f964a80267aabfd2c6b6e4b7c233572829566d59fb2b98af3f36d05f29e4"
May 28 08:32:09.514313 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:32:09.514293    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1496f964a80267aabfd2c6b6e4b7c233572829566d59fb2b98af3f36d05f29e4\": container with ID starting with 1496f964a80267aabfd2c6b6e4b7c233572829566d59fb2b98af3f36d05f29e4 not found: ID does not exist" containerID="1496f964a80267aabfd2c6b6e4b7c233572829566d59fb2b98af3f36d05f29e4"
May 28 08:32:09.514358 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.514321    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1496f964a80267aabfd2c6b6e4b7c233572829566d59fb2b98af3f36d05f29e4"} err="failed to get container status \"1496f964a80267aabfd2c6b6e4b7c233572829566d59fb2b98af3f36d05f29e4\": rpc error: code = NotFound desc = could not find container \"1496f964a80267aabfd2c6b6e4b7c233572829566d59fb2b98af3f36d05f29e4\": container with ID starting with 1496f964a80267aabfd2c6b6e4b7c233572829566d59fb2b98af3f36d05f29e4 not found: ID does not exist"
May 28 08:32:09.514358 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.514339    2551 scope.go:117] "RemoveContainer" containerID="6e5f74435c0c31ba37422900b7a6b2d7ffa81e3c0f51e0df691f915514fb82e7"
May 28 08:32:09.514556 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:32:09.514541    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5f74435c0c31ba37422900b7a6b2d7ffa81e3c0f51e0df691f915514fb82e7\": container with ID starting with 6e5f74435c0c31ba37422900b7a6b2d7ffa81e3c0f51e0df691f915514fb82e7 not found: ID does not exist" containerID="6e5f74435c0c31ba37422900b7a6b2d7ffa81e3c0f51e0df691f915514fb82e7"
May 28 08:32:09.514595 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.514558    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5f74435c0c31ba37422900b7a6b2d7ffa81e3c0f51e0df691f915514fb82e7"} err="failed to get container status \"6e5f74435c0c31ba37422900b7a6b2d7ffa81e3c0f51e0df691f915514fb82e7\": rpc error: code = NotFound desc = could not find container \"6e5f74435c0c31ba37422900b7a6b2d7ffa81e3c0f51e0df691f915514fb82e7\": container with ID starting with 6e5f74435c0c31ba37422900b7a6b2d7ffa81e3c0f51e0df691f915514fb82e7 not found: ID does not exist"
May 28 08:32:09.514595 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.514571    2551 scope.go:117] "RemoveContainer" containerID="5ebe51e5633ff389a26803a249f1ff04f33db4de224e56a036e322e774bc803c"
May 28 08:32:09.514768 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:32:09.514750    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ebe51e5633ff389a26803a249f1ff04f33db4de224e56a036e322e774bc803c\": container with ID starting with 5ebe51e5633ff389a26803a249f1ff04f33db4de224e56a036e322e774bc803c not found: ID does not exist" containerID="5ebe51e5633ff389a26803a249f1ff04f33db4de224e56a036e322e774bc803c"
May 28 08:32:09.514810 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.514772    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebe51e5633ff389a26803a249f1ff04f33db4de224e56a036e322e774bc803c"} err="failed to get container status \"5ebe51e5633ff389a26803a249f1ff04f33db4de224e56a036e322e774bc803c\": rpc error: code = NotFound desc = could not find container \"5ebe51e5633ff389a26803a249f1ff04f33db4de224e56a036e322e774bc803c\": container with ID starting with 5ebe51e5633ff389a26803a249f1ff04f33db4de224e56a036e322e774bc803c not found: ID does not exist"
May 28 08:32:09.514810 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.514783    2551 scope.go:117] "RemoveContainer" containerID="e94bda03810502285373a8c5fbda1cac5ee43ff6d9613f3e4b0c4c250ea23f88"
May 28 08:32:09.514993 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:32:09.514979    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94bda03810502285373a8c5fbda1cac5ee43ff6d9613f3e4b0c4c250ea23f88\": container with ID starting with e94bda03810502285373a8c5fbda1cac5ee43ff6d9613f3e4b0c4c250ea23f88 not found: ID does not exist" containerID="e94bda03810502285373a8c5fbda1cac5ee43ff6d9613f3e4b0c4c250ea23f88"
May 28 08:32:09.515037 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.514997    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94bda03810502285373a8c5fbda1cac5ee43ff6d9613f3e4b0c4c250ea23f88"} err="failed to get container status \"e94bda03810502285373a8c5fbda1cac5ee43ff6d9613f3e4b0c4c250ea23f88\": rpc error: code = NotFound desc = could not find container \"e94bda03810502285373a8c5fbda1cac5ee43ff6d9613f3e4b0c4c250ea23f88\": container with ID starting with e94bda03810502285373a8c5fbda1cac5ee43ff6d9613f3e4b0c4c250ea23f88 not found: ID does not exist"
May 28 08:32:09.515037 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.515008    2551 scope.go:117] "RemoveContainer" containerID="53a6f59e1a40a09c41ae714cdabe5fd0197b284470960a84257935f56516e82c"
May 28 08:32:09.515173 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:32:09.515155    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a6f59e1a40a09c41ae714cdabe5fd0197b284470960a84257935f56516e82c\": container with ID starting with 53a6f59e1a40a09c41ae714cdabe5fd0197b284470960a84257935f56516e82c not found: ID does not exist" containerID="53a6f59e1a40a09c41ae714cdabe5fd0197b284470960a84257935f56516e82c"
May 28 08:32:09.515217 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.515176    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a6f59e1a40a09c41ae714cdabe5fd0197b284470960a84257935f56516e82c"} err="failed to get container status \"53a6f59e1a40a09c41ae714cdabe5fd0197b284470960a84257935f56516e82c\": rpc error: code = NotFound desc = could not find container \"53a6f59e1a40a09c41ae714cdabe5fd0197b284470960a84257935f56516e82c\": container with ID starting with 53a6f59e1a40a09c41ae714cdabe5fd0197b284470960a84257935f56516e82c not found: ID does not exist"
May 28 08:32:09.557819 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.557782    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7tbb\" (UniqueName: \"kubernetes.io/projected/3eb81d49-4db2-4ab9-b75f-01dd473a0389-kube-api-access-l7tbb\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:32:09.557819 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.557818    2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-online-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:32:09.557819 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.557829    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-registry-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:32:09.558016 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.557845    2551 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-ui-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:32:09.558016 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.557853    2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb81d49-4db2-4ab9-b75f-01dd473a0389-feast-data\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:32:09.558016 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.557861    2551 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3eb81d49-4db2-4ab9-b75f-01dd473a0389-offline-tls\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:32:09.748423 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.748328    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"]
May 28 08:32:09.751791 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:09.751765    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-5d7xz"]
May 28 08:32:10.195273 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:10.195195    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" path="/var/lib/kubelet/pods/0c202a7e-f36e-4467-8473-ffc852dc6f0d/volumes"
May 28 08:32:10.195750 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:10.195737    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" path="/var/lib/kubelet/pods/3eb81d49-4db2-4ab9-b75f-01dd473a0389/volumes"
May 28 08:32:27.906516 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906480    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9h5r8/must-gather-58q5k"]
May 28 08:32:27.906883 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906805    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="registry"
May 28 08:32:27.906883 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906817    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="registry"
May 28 08:32:27.906883 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906826    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="offline"
May 28 08:32:27.906883 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906832    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="offline"
May 28 08:32:27.906883 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906841    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="online"
May 28 08:32:27.906883 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906848    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="online"
May 28 08:32:27.906883 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906853    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="feast-apply"
May 28 08:32:27.906883 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906858    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="feast-apply"
May 28 08:32:27.906883 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906865    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="feast-init"
May 28 08:32:27.906883 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906870    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="feast-init"
May 28 08:32:27.906883 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906877    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="ui"
May 28 08:32:27.906883 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906882    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="ui"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906891    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="offline"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906896    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="offline"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906905    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="feast-init"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906909    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="feast-init"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906919    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="online"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906924    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="online"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906933    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="feast-apply"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906938    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="feast-apply"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906943    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="ui"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906947    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="ui"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906990    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="online"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.906997    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="online"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.907002    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="registry"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.907009    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="ui"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.907015    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="ui"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.907022    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="0c202a7e-f36e-4467-8473-ffc852dc6f0d" containerName="offline"
May 28 08:32:27.907247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.907028    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="3eb81d49-4db2-4ab9-b75f-01dd473a0389" containerName="offline"
May 28 08:32:27.911982 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.911965    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9h5r8/must-gather-58q5k"
May 28 08:32:27.914743 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.914715    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9h5r8\"/\"kube-root-ca.crt\""
May 28 08:32:27.914879 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.914715    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9h5r8\"/\"openshift-service-ca.crt\""
May 28 08:32:27.915004 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.914987    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9h5r8\"/\"default-dockercfg-qt775\""
May 28 08:32:27.915566 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:27.915544    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9h5r8/must-gather-58q5k"]
May 28 08:32:28.008777 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:28.008744    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5mx8\" (UniqueName: \"kubernetes.io/projected/e23ade72-7d7b-4670-9d1e-ebd500e0a7c5-kube-api-access-d5mx8\") pod \"must-gather-58q5k\" (UID: \"e23ade72-7d7b-4670-9d1e-ebd500e0a7c5\") " pod="openshift-must-gather-9h5r8/must-gather-58q5k"
May 28 08:32:28.008920 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:28.008787    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e23ade72-7d7b-4670-9d1e-ebd500e0a7c5-must-gather-output\") pod \"must-gather-58q5k\" (UID: \"e23ade72-7d7b-4670-9d1e-ebd500e0a7c5\") " pod="openshift-must-gather-9h5r8/must-gather-58q5k"
May 28 08:32:28.109849 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:28.109813    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d5mx8\" (UniqueName: \"kubernetes.io/projected/e23ade72-7d7b-4670-9d1e-ebd500e0a7c5-kube-api-access-d5mx8\") pod \"must-gather-58q5k\" (UID: \"e23ade72-7d7b-4670-9d1e-ebd500e0a7c5\") " pod="openshift-must-gather-9h5r8/must-gather-58q5k"
May 28 08:32:28.110010 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:28.109860    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e23ade72-7d7b-4670-9d1e-ebd500e0a7c5-must-gather-output\") pod \"must-gather-58q5k\" (UID: \"e23ade72-7d7b-4670-9d1e-ebd500e0a7c5\") " pod="openshift-must-gather-9h5r8/must-gather-58q5k"
May 28 08:32:28.110151 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:28.110136    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e23ade72-7d7b-4670-9d1e-ebd500e0a7c5-must-gather-output\") pod \"must-gather-58q5k\" (UID: \"e23ade72-7d7b-4670-9d1e-ebd500e0a7c5\") " pod="openshift-must-gather-9h5r8/must-gather-58q5k"
May 28 08:32:28.118153 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:28.118126    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5mx8\" (UniqueName: \"kubernetes.io/projected/e23ade72-7d7b-4670-9d1e-ebd500e0a7c5-kube-api-access-d5mx8\") pod \"must-gather-58q5k\" (UID: \"e23ade72-7d7b-4670-9d1e-ebd500e0a7c5\") " pod="openshift-must-gather-9h5r8/must-gather-58q5k"
May 28 08:32:28.231725 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:28.231699    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9h5r8/must-gather-58q5k"
May 28 08:32:28.351035 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:28.351005    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9h5r8/must-gather-58q5k"]
May 28 08:32:28.353781 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:32:28.353744    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode23ade72_7d7b_4670_9d1e_ebd500e0a7c5.slice/crio-f840688efcc5123f27309b24242d998aac46989fdb1d35f62eb9ad1b95e4cf94 WatchSource:0}: Error finding container f840688efcc5123f27309b24242d998aac46989fdb1d35f62eb9ad1b95e4cf94: Status 404 returned error can't find the container with id f840688efcc5123f27309b24242d998aac46989fdb1d35f62eb9ad1b95e4cf94
May 28 08:32:28.485724 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:28.485650    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9h5r8/must-gather-58q5k" event={"ID":"e23ade72-7d7b-4670-9d1e-ebd500e0a7c5","Type":"ContainerStarted","Data":"f840688efcc5123f27309b24242d998aac46989fdb1d35f62eb9ad1b95e4cf94"}
May 28 08:32:35.511301 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:35.511259    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9h5r8/must-gather-58q5k" event={"ID":"e23ade72-7d7b-4670-9d1e-ebd500e0a7c5","Type":"ContainerStarted","Data":"2aff8fe0eb41a4ec1835af0bc74f3e311e4367e9525f790b8645954f08f64959"}
May 28 08:32:35.511301 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:35.511299    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9h5r8/must-gather-58q5k" event={"ID":"e23ade72-7d7b-4670-9d1e-ebd500e0a7c5","Type":"ContainerStarted","Data":"60789aa4558dab203921f843ca7394e49722fd242f154489e4d8f3ed54d8e707"}
May 28 08:32:35.528238 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:35.528182    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-9h5r8/must-gather-58q5k" podStartSLOduration=2.099938003 podStartE2EDuration="8.528169224s" podCreationTimestamp="2026-05-28 08:32:27 +0000 UTC" firstStartedPulling="2026-05-28 08:32:28.355781696 +0000 UTC m=+2000.795853964" lastFinishedPulling="2026-05-28 08:32:34.784012915 +0000 UTC m=+2007.224085185" observedRunningTime="2026-05-28 08:32:35.527599114 +0000 UTC m=+2007.967671404" watchObservedRunningTime="2026-05-28 08:32:35.528169224 +0000 UTC m=+2007.968241512"
May 28 08:32:44.540275 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:44.540239    2551 generic.go:358] "Generic (PLEG): container finished" podID="e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" containerID="60789aa4558dab203921f843ca7394e49722fd242f154489e4d8f3ed54d8e707" exitCode=0
May 28 08:32:44.540705 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:44.540310    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9h5r8/must-gather-58q5k" event={"ID":"e23ade72-7d7b-4670-9d1e-ebd500e0a7c5","Type":"ContainerDied","Data":"60789aa4558dab203921f843ca7394e49722fd242f154489e4d8f3ed54d8e707"}
May 28 08:32:44.540705 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:44.540633    2551 scope.go:117] "RemoveContainer" containerID="60789aa4558dab203921f843ca7394e49722fd242f154489e4d8f3ed54d8e707"
May 28 08:32:45.331089 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:45.331055    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9h5r8_must-gather-58q5k_e23ade72-7d7b-4670-9d1e-ebd500e0a7c5/gather/0.log"
May 28 08:32:48.271475 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:48.271443    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cnjvq_a659431e-376b-4589-8ae4-27694a9b763f/global-pull-secret-syncer/0.log"
May 28 08:32:48.437936 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:48.437907    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nfsn2_a310a270-ab14-4bfc-ade4-123b9ee11db4/konnectivity-agent/0.log"
May 28 08:32:48.485725 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:48.485695    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-248.ec2.internal_fc0a0cad24c1a6672f2575bc0047f126/haproxy/0.log"
May 28 08:32:50.660650 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:50.660617    2551 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-9h5r8/must-gather-58q5k"]
May 28 08:32:50.661041 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:50.660819    2551 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-9h5r8/must-gather-58q5k" podUID="e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" containerName="copy" containerID="cri-o://2aff8fe0eb41a4ec1835af0bc74f3e311e4367e9525f790b8645954f08f64959" gracePeriod=2
May 28 08:32:50.663004 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:50.662970    2551 status_manager.go:895] "Failed to get status for pod" podUID="e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" pod="openshift-must-gather-9h5r8/must-gather-58q5k" err="pods \"must-gather-58q5k\" is forbidden: User \"system:node:ip-10-0-136-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-9h5r8\": no relationship found between node 'ip-10-0-136-248.ec2.internal' and this object"
May 28 08:32:50.665059 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:50.664472    2551 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-9h5r8/must-gather-58q5k"]
May 28 08:32:50.889108 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:50.889088    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9h5r8_must-gather-58q5k_e23ade72-7d7b-4670-9d1e-ebd500e0a7c5/copy/0.log"
May 28 08:32:50.889458 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:50.889443    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9h5r8/must-gather-58q5k"
May 28 08:32:50.891504 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:50.891481    2551 status_manager.go:895] "Failed to get status for pod" podUID="e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" pod="openshift-must-gather-9h5r8/must-gather-58q5k" err="pods \"must-gather-58q5k\" is forbidden: User \"system:node:ip-10-0-136-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-9h5r8\": no relationship found between node 'ip-10-0-136-248.ec2.internal' and this object"
May 28 08:32:51.008846 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.008817    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e23ade72-7d7b-4670-9d1e-ebd500e0a7c5-must-gather-output\") pod \"e23ade72-7d7b-4670-9d1e-ebd500e0a7c5\" (UID: \"e23ade72-7d7b-4670-9d1e-ebd500e0a7c5\") "
May 28 08:32:51.008983 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.008938    2551 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5mx8\" (UniqueName: \"kubernetes.io/projected/e23ade72-7d7b-4670-9d1e-ebd500e0a7c5-kube-api-access-d5mx8\") pod \"e23ade72-7d7b-4670-9d1e-ebd500e0a7c5\" (UID: \"e23ade72-7d7b-4670-9d1e-ebd500e0a7c5\") "
May 28 08:32:51.009210 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.009188    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23ade72-7d7b-4670-9d1e-ebd500e0a7c5-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" (UID: "e23ade72-7d7b-4670-9d1e-ebd500e0a7c5"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 28 08:32:51.011147 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.011119    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23ade72-7d7b-4670-9d1e-ebd500e0a7c5-kube-api-access-d5mx8" (OuterVolumeSpecName: "kube-api-access-d5mx8") pod "e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" (UID: "e23ade72-7d7b-4670-9d1e-ebd500e0a7c5"). InnerVolumeSpecName "kube-api-access-d5mx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 28 08:32:51.109658 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.109629    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5mx8\" (UniqueName: \"kubernetes.io/projected/e23ade72-7d7b-4670-9d1e-ebd500e0a7c5-kube-api-access-d5mx8\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:32:51.109658 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.109656    2551 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e23ade72-7d7b-4670-9d1e-ebd500e0a7c5-must-gather-output\") on node \"ip-10-0-136-248.ec2.internal\" DevicePath \"\""
May 28 08:32:51.550331 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.550295    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a17ff24-aaf1-4538-b12e-84eb498faff3/alertmanager/0.log"
May 28 08:32:51.562177 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.562156    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9h5r8_must-gather-58q5k_e23ade72-7d7b-4670-9d1e-ebd500e0a7c5/copy/0.log"
May 28 08:32:51.562483 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.562462    2551 generic.go:358] "Generic (PLEG): container finished" podID="e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" containerID="2aff8fe0eb41a4ec1835af0bc74f3e311e4367e9525f790b8645954f08f64959" exitCode=143
May 28 08:32:51.562560 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.562525    2551 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9h5r8/must-gather-58q5k"
May 28 08:32:51.562620 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.562560    2551 scope.go:117] "RemoveContainer" containerID="2aff8fe0eb41a4ec1835af0bc74f3e311e4367e9525f790b8645954f08f64959"
May 28 08:32:51.565409 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.564830    2551 status_manager.go:895] "Failed to get status for pod" podUID="e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" pod="openshift-must-gather-9h5r8/must-gather-58q5k" err="pods \"must-gather-58q5k\" is forbidden: User \"system:node:ip-10-0-136-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-9h5r8\": no relationship found between node 'ip-10-0-136-248.ec2.internal' and this object"
May 28 08:32:51.574937 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.574919    2551 scope.go:117] "RemoveContainer" containerID="60789aa4558dab203921f843ca7394e49722fd242f154489e4d8f3ed54d8e707"
May 28 08:32:51.575564 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.575539    2551 status_manager.go:895] "Failed to get status for pod" podUID="e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" pod="openshift-must-gather-9h5r8/must-gather-58q5k" err="pods \"must-gather-58q5k\" is forbidden: User \"system:node:ip-10-0-136-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-9h5r8\": no relationship found between node 'ip-10-0-136-248.ec2.internal' and this object"
May 28 08:32:51.576376 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.576361    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a17ff24-aaf1-4538-b12e-84eb498faff3/config-reloader/0.log"
May 28 08:32:51.586394 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.586355    2551 scope.go:117] "RemoveContainer" containerID="2aff8fe0eb41a4ec1835af0bc74f3e311e4367e9525f790b8645954f08f64959"
May 28 08:32:51.586660 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:32:51.586642    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aff8fe0eb41a4ec1835af0bc74f3e311e4367e9525f790b8645954f08f64959\": container with ID starting with 2aff8fe0eb41a4ec1835af0bc74f3e311e4367e9525f790b8645954f08f64959 not found: ID does not exist" containerID="2aff8fe0eb41a4ec1835af0bc74f3e311e4367e9525f790b8645954f08f64959"
May 28 08:32:51.586718 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.586668    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aff8fe0eb41a4ec1835af0bc74f3e311e4367e9525f790b8645954f08f64959"} err="failed to get container status \"2aff8fe0eb41a4ec1835af0bc74f3e311e4367e9525f790b8645954f08f64959\": rpc error: code = NotFound desc = could not find container \"2aff8fe0eb41a4ec1835af0bc74f3e311e4367e9525f790b8645954f08f64959\": container with ID starting with 2aff8fe0eb41a4ec1835af0bc74f3e311e4367e9525f790b8645954f08f64959 not found: ID does not exist"
May 28 08:32:51.586718 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.586686    2551 scope.go:117] "RemoveContainer" containerID="60789aa4558dab203921f843ca7394e49722fd242f154489e4d8f3ed54d8e707"
May 28 08:32:51.586930 ip-10-0-136-248 kubenswrapper[2551]: E0528 08:32:51.586909    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60789aa4558dab203921f843ca7394e49722fd242f154489e4d8f3ed54d8e707\": container with ID starting with 60789aa4558dab203921f843ca7394e49722fd242f154489e4d8f3ed54d8e707 not found: ID does not exist" containerID="60789aa4558dab203921f843ca7394e49722fd242f154489e4d8f3ed54d8e707"
May 28 08:32:51.586980 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.586937    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60789aa4558dab203921f843ca7394e49722fd242f154489e4d8f3ed54d8e707"} err="failed to get container status \"60789aa4558dab203921f843ca7394e49722fd242f154489e4d8f3ed54d8e707\": rpc error: code = NotFound desc = could not find container \"60789aa4558dab203921f843ca7394e49722fd242f154489e4d8f3ed54d8e707\": container with ID starting with 60789aa4558dab203921f843ca7394e49722fd242f154489e4d8f3ed54d8e707 not found: ID does not exist"
May 28 08:32:51.609106 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.609082    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a17ff24-aaf1-4538-b12e-84eb498faff3/kube-rbac-proxy-web/0.log"
May 28 08:32:51.638839 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.638816    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a17ff24-aaf1-4538-b12e-84eb498faff3/kube-rbac-proxy/0.log"
May 28 08:32:51.668070 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.668035    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a17ff24-aaf1-4538-b12e-84eb498faff3/kube-rbac-proxy-metric/0.log"
May 28 08:32:51.695447 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.695421    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a17ff24-aaf1-4538-b12e-84eb498faff3/prom-label-proxy/0.log"
May 28 08:32:51.729111 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.729085    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2a17ff24-aaf1-4538-b12e-84eb498faff3/init-config-reloader/0.log"
May 28 08:32:51.777479 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.777455    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-7q7sh_c7cf5bb2-5b95-4887-8fef-dbe7bdfd2ce4/cluster-monitoring-operator/0.log"
May 28 08:32:51.961106 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.961061    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fdhkw_30da5bc8-4838-4345-a058-65b26ae80ec8/node-exporter/0.log"
May 28 08:32:51.984236 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:51.984210    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fdhkw_30da5bc8-4838-4345-a058-65b26ae80ec8/kube-rbac-proxy/0.log"
May 28 08:32:52.007812 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:52.007793    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fdhkw_30da5bc8-4838-4345-a058-65b26ae80ec8/init-textfile/0.log"
May 28 08:32:52.187225 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:52.187182    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-dzv5g_a9251b57-4aa4-455d-9548-4aa1de30f61d/kube-rbac-proxy-main/0.log"
May 28 08:32:52.194615 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:52.194579    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" path="/var/lib/kubelet/pods/e23ade72-7d7b-4670-9d1e-ebd500e0a7c5/volumes"
May 28 08:32:52.209256 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:52.209237    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-dzv5g_a9251b57-4aa4-455d-9548-4aa1de30f61d/kube-rbac-proxy-self/0.log"
May 28 08:32:52.233715 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:52.233659    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-dzv5g_a9251b57-4aa4-455d-9548-4aa1de30f61d/openshift-state-metrics/0.log"
May 28 08:32:52.496840 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:52.496764    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cc545fc4-mhqk4_17e2c6ff-5f87-406b-ae23-e8fbace620f4/telemeter-client/0.log"
May 28 08:32:52.521471 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:52.521445    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cc545fc4-mhqk4_17e2c6ff-5f87-406b-ae23-e8fbace620f4/reload/0.log"
May 28 08:32:52.543163 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:52.543125    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cc545fc4-mhqk4_17e2c6ff-5f87-406b-ae23-e8fbace620f4/kube-rbac-proxy/0.log"
May 28 08:32:52.574673 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:52.574626    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb9978d9c-gn2xj_6f51145c-9285-4e23-9143-5a9e983d98b0/thanos-query/0.log"
May 28 08:32:52.596772 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:52.596746    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb9978d9c-gn2xj_6f51145c-9285-4e23-9143-5a9e983d98b0/kube-rbac-proxy-web/0.log"
May 28 08:32:52.619059 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:52.619030    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb9978d9c-gn2xj_6f51145c-9285-4e23-9143-5a9e983d98b0/kube-rbac-proxy/0.log"
May 28 08:32:52.640341 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:52.640315    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb9978d9c-gn2xj_6f51145c-9285-4e23-9143-5a9e983d98b0/prom-label-proxy/0.log"
May 28 08:32:52.663320 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:52.663298    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb9978d9c-gn2xj_6f51145c-9285-4e23-9143-5a9e983d98b0/kube-rbac-proxy-rules/0.log"
May 28 08:32:52.687072 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:52.687050    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb9978d9c-gn2xj_6f51145c-9285-4e23-9143-5a9e983d98b0/kube-rbac-proxy-metrics/0.log"
May 28 08:32:54.215159 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.215131    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/1.log"
May 28 08:32:54.219276 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.219242    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-t28bq_6578639e-59ec-4e29-881d-f6d789b54b8f/console-operator/2.log"
May 28 08:32:54.531071 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.530982    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b566cf6bf-nxxhf_6cb634ee-00b2-4eb2-9092-acc963210689/console/0.log"
May 28 08:32:54.558717 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.558689    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-597rs_8a78dae7-58a6-46a6-b6fa-c46e0df30aba/download-server/0.log"
May 28 08:32:54.823976 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.823896    2551 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"]
May 28 08:32:54.824208 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.824196    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" containerName="gather"
May 28 08:32:54.824257 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.824209    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" containerName="gather"
May 28 08:32:54.824257 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.824221    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" containerName="copy"
May 28 08:32:54.824257 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.824226    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" containerName="copy"
May 28 08:32:54.824346 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.824297    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" containerName="copy"
May 28 08:32:54.824346 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.824308    2551 memory_manager.go:356] "RemoveStaleState removing state" podUID="e23ade72-7d7b-4670-9d1e-ebd500e0a7c5" containerName="gather"
May 28 08:32:54.827334 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.827312    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:54.833537 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.833493    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j9p4l\"/\"kube-root-ca.crt\""
May 28 08:32:54.834320 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.834305    2551 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j9p4l\"/\"openshift-service-ca.crt\""
May 28 08:32:54.835063 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.835048    2551 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j9p4l\"/\"default-dockercfg-6mk6k\""
May 28 08:32:54.876491 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.876462    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"]
May 28 08:32:54.943033 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.943002    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d6c49a73-d7cb-4a27-90c6-ea9eb32c2391-podres\") pod \"perf-node-gather-daemonset-82vdt\" (UID: \"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391\") " pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:54.943161 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.943045    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwx8k\" (UniqueName: \"kubernetes.io/projected/d6c49a73-d7cb-4a27-90c6-ea9eb32c2391-kube-api-access-hwx8k\") pod \"perf-node-gather-daemonset-82vdt\" (UID: \"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391\") " pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:54.943161 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.943123    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d6c49a73-d7cb-4a27-90c6-ea9eb32c2391-proc\") pod \"perf-node-gather-daemonset-82vdt\" (UID: \"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391\") " pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:54.943247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.943170    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6c49a73-d7cb-4a27-90c6-ea9eb32c2391-lib-modules\") pod \"perf-node-gather-daemonset-82vdt\" (UID: \"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391\") " pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:54.943247 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.943196    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6c49a73-d7cb-4a27-90c6-ea9eb32c2391-sys\") pod \"perf-node-gather-daemonset-82vdt\" (UID: \"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391\") " pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:54.988558 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:54.988527    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-6qh4c_a88b511e-eba6-4500-99f9-021664212882/volume-data-source-validator/0.log"
May 28 08:32:55.044580 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.044552    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6c49a73-d7cb-4a27-90c6-ea9eb32c2391-lib-modules\") pod \"perf-node-gather-daemonset-82vdt\" (UID: \"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391\") " pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:55.044716 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.044585    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6c49a73-d7cb-4a27-90c6-ea9eb32c2391-sys\") pod \"perf-node-gather-daemonset-82vdt\" (UID: \"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391\") " pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:55.044716 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.044644    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d6c49a73-d7cb-4a27-90c6-ea9eb32c2391-podres\") pod \"perf-node-gather-daemonset-82vdt\" (UID: \"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391\") " pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:55.044716 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.044677    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hwx8k\" (UniqueName: \"kubernetes.io/projected/d6c49a73-d7cb-4a27-90c6-ea9eb32c2391-kube-api-access-hwx8k\") pod \"perf-node-gather-daemonset-82vdt\" (UID: \"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391\") " pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:55.044865 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.044730    2551 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d6c49a73-d7cb-4a27-90c6-ea9eb32c2391-proc\") pod \"perf-node-gather-daemonset-82vdt\" (UID: \"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391\") " pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:55.044865 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.044758    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6c49a73-d7cb-4a27-90c6-ea9eb32c2391-sys\") pod \"perf-node-gather-daemonset-82vdt\" (UID: \"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391\") " pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:55.044865 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.044732    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6c49a73-d7cb-4a27-90c6-ea9eb32c2391-lib-modules\") pod \"perf-node-gather-daemonset-82vdt\" (UID: \"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391\") " pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:55.044865 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.044796    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d6c49a73-d7cb-4a27-90c6-ea9eb32c2391-podres\") pod \"perf-node-gather-daemonset-82vdt\" (UID: \"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391\") " pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:55.044865 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.044811    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d6c49a73-d7cb-4a27-90c6-ea9eb32c2391-proc\") pod \"perf-node-gather-daemonset-82vdt\" (UID: \"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391\") " pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:55.053074 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.053055    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwx8k\" (UniqueName: \"kubernetes.io/projected/d6c49a73-d7cb-4a27-90c6-ea9eb32c2391-kube-api-access-hwx8k\") pod \"perf-node-gather-daemonset-82vdt\" (UID: \"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391\") " pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:55.136898 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.136812    2551 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:55.259541 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.259516    2551 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"]
May 28 08:32:55.262303 ip-10-0-136-248 kubenswrapper[2551]: W0528 08:32:55.262275    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd6c49a73_d7cb_4a27_90c6_ea9eb32c2391.slice/crio-b1915895ff5045ceea7a812cb8afb127963547e0aa9b12be7ee9a47dbb0ac8f8 WatchSource:0}: Error finding container b1915895ff5045ceea7a812cb8afb127963547e0aa9b12be7ee9a47dbb0ac8f8: Status 404 returned error can't find the container with id b1915895ff5045ceea7a812cb8afb127963547e0aa9b12be7ee9a47dbb0ac8f8
May 28 08:32:55.577453 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.577421    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt" event={"ID":"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391","Type":"ContainerStarted","Data":"29a95af34606e42d57e5af7c6f055eea82b6985ae6819f8d27984834778d845a"}
May 28 08:32:55.577453 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.577456    2551 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt" event={"ID":"d6c49a73-d7cb-4a27-90c6-ea9eb32c2391","Type":"ContainerStarted","Data":"b1915895ff5045ceea7a812cb8afb127963547e0aa9b12be7ee9a47dbb0ac8f8"}
May 28 08:32:55.577675 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.577554    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:32:55.593483 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.593441    2551 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt" podStartSLOduration=1.593428501 podStartE2EDuration="1.593428501s" podCreationTimestamp="2026-05-28 08:32:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 08:32:55.59112182 +0000 UTC m=+2028.031194108" watchObservedRunningTime="2026-05-28 08:32:55.593428501 +0000 UTC m=+2028.033500790"
May 28 08:32:55.667751 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.667728    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q99nf_aaa6c56b-06fe-483b-9755-25d6f4930f41/dns/0.log"
May 28 08:32:55.686691 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.686666    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q99nf_aaa6c56b-06fe-483b-9755-25d6f4930f41/kube-rbac-proxy/0.log"
May 28 08:32:55.707035 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:55.707012    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8vcnc_91168ae0-b0ae-482d-be58-f879dcae55f1/dns-node-resolver/0.log"
May 28 08:32:56.113987 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:56.113956    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-66575d684f-4nkxt_bc35ffa7-5b6f-45dd-93d5-5894c38b5259/registry/0.log"
May 28 08:32:56.152103 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:56.152073    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q9tbf_49ad95da-9ff8-462b-86aa-6f6eedf9ae3e/node-ca/0.log"
May 28 08:32:56.774393 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:56.774367    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56dd9859f6-76gdf_0039c6a2-d3a2-40aa-9799-8237db852588/router/0.log"
May 28 08:32:57.084195 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:57.084082    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cpkhd_483c9981-b25f-41e0-92e1-d8ebcf6d6966/serve-healthcheck-canary/0.log"
May 28 08:32:57.554930 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:57.554878    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xv4mt_0291d14c-00af-44c4-b903-6d02d1daa8be/kube-rbac-proxy/0.log"
May 28 08:32:57.573003 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:57.572975    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xv4mt_0291d14c-00af-44c4-b903-6d02d1daa8be/exporter/0.log"
May 28 08:32:57.594089 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:32:57.594064    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xv4mt_0291d14c-00af-44c4-b903-6d02d1daa8be/extractor/0.log"
May 28 08:33:01.590731 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:01.590706    2551 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j9p4l/perf-node-gather-daemonset-82vdt"
May 28 08:33:02.825776 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:02.825751    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wcbtq_64ffc18e-c32d-4d14-869f-fe837ec83037/kube-multus-additional-cni-plugins/0.log"
May 28 08:33:02.844968 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:02.844943    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wcbtq_64ffc18e-c32d-4d14-869f-fe837ec83037/egress-router-binary-copy/0.log"
May 28 08:33:02.864849 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:02.864825    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wcbtq_64ffc18e-c32d-4d14-869f-fe837ec83037/cni-plugins/0.log"
May 28 08:33:02.884090 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:02.884068    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wcbtq_64ffc18e-c32d-4d14-869f-fe837ec83037/bond-cni-plugin/0.log"
May 28 08:33:02.904935 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:02.904912    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wcbtq_64ffc18e-c32d-4d14-869f-fe837ec83037/routeoverride-cni/0.log"
May 28 08:33:02.926631 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:02.926609    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wcbtq_64ffc18e-c32d-4d14-869f-fe837ec83037/whereabouts-cni-bincopy/0.log"
May 28 08:33:02.945104 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:02.945081    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wcbtq_64ffc18e-c32d-4d14-869f-fe837ec83037/whereabouts-cni/0.log"
May 28 08:33:02.978652 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:02.978630    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fjgcp_7bce0101-b6a3-4da0-bab9-10edb7bcad0b/kube-multus/0.log"
May 28 08:33:03.055463 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:03.055438    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dfswh_2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd/network-metrics-daemon/0.log"
May 28 08:33:03.073066 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:03.073047    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dfswh_2f629a2a-caeb-4e3c-8dfb-b835ecd14fbd/kube-rbac-proxy/0.log"
May 28 08:33:04.524901 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:04.524872    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vvpbp_520071c5-2f56-457c-9fc5-d833415d3248/ovn-controller/0.log"
May 28 08:33:04.551062 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:04.551030    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vvpbp_520071c5-2f56-457c-9fc5-d833415d3248/ovn-acl-logging/0.log"
May 28 08:33:04.570178 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:04.570156    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vvpbp_520071c5-2f56-457c-9fc5-d833415d3248/kube-rbac-proxy-node/0.log"
May 28 08:33:04.593081 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:04.592993    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vvpbp_520071c5-2f56-457c-9fc5-d833415d3248/kube-rbac-proxy-ovn-metrics/0.log"
May 28 08:33:04.613555 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:04.613527    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vvpbp_520071c5-2f56-457c-9fc5-d833415d3248/northd/0.log"
May 28 08:33:04.634448 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:04.634430    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vvpbp_520071c5-2f56-457c-9fc5-d833415d3248/nbdb/0.log"
May 28 08:33:04.655552 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:04.655529    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vvpbp_520071c5-2f56-457c-9fc5-d833415d3248/sbdb/0.log"
May 28 08:33:04.749645 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:04.749617    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vvpbp_520071c5-2f56-457c-9fc5-d833415d3248/ovnkube-controller/0.log"
May 28 08:33:05.737051 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:05.737022    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-t782k_b33f26fc-296c-4693-b53a-a88ccd6f2236/network-check-target-container/0.log"
May 28 08:33:06.562043 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:06.562016    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-p6zng_1fe5d6a6-0c2f-4e92-a246-e92a2f6f14a9/iptables-alerter/0.log"
May 28 08:33:07.106180 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:07.106152    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-lfc7s_ae6d6f2a-72d0-44f2-b512-e739ccf792eb/tuned/0.log"
May 28 08:33:09.790884 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:09.790849    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-67d667c7d-5xvxd_29cb26de-9ec3-4170-a868-4b5b704f1e93/service-ca-controller/0.log"
May 28 08:33:10.184319 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:10.184288    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-mq2lj_6fcecffb-0699-457c-a17f-be7801546dd8/csi-driver/0.log"
May 28 08:33:10.206075 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:10.206048    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-mq2lj_6fcecffb-0699-457c-a17f-be7801546dd8/csi-node-driver-registrar/0.log"
May 28 08:33:10.224474 ip-10-0-136-248 kubenswrapper[2551]: I0528 08:33:10.224449    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-mq2lj_6fcecffb-0699-457c-a17f-be7801546dd8/csi-liveness-probe/0.log"